You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Pa Rö <pa...@googlemail.com> on 2015/08/02 16:04:25 UTC

spark no output

hi community,

i have run my k-means spark application on 1million data points. the
program works, but no output in the hdfs is generated. when it runs on
10.000 points, a output is written.

maybe someone has an idea?

best regards,
paul

Re: spark no output

Posted by Todd Nist <ts...@gmail.com>.

I appears as though the Spark application exceed the amount of memory for a
given task in yarn:

15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN
for exceeding memory limits. 7.0 GB of 7 GB physical memory used.
Consider boosting spark.yarn.executor.memoryOverhead.

You may want to look at adjusting the memory of Spark and tuning the
following:

spark.yarn.executor.memoryOverhead
spark.storage.memoryFraction
spark.storage.safetyFraction

Good read on this here:

http://www.wdong.org/wordpress/blog/2015/01/08/spark-on-yarn-where-have-all-my-memory-gone/

And of course in the spark docs as well:

Property NameDefaultMeaningspark.yarn.am.memory512mAmount of memory to use
for the YARN Application Master in client mode, in the same format as JVM
memory strings (e.g. 512m, 2g). In cluster mode, use spark.driver.memory
 instead.spark.yarn.executor.memoryOverheadexecutorMemory * 0.10, with
minimum of 384The amount of off heap memory (in megabytes) to be allocated
per executor. This is memory that accounts for things like VM overheads,
interned strings, other native overheads, etc. This tends to grow with the
executor size (typically 6-10%).spark.yarn.driver.memoryOverheaddriverMemory
* 0.07, with minimum of 384The amount of off heap memory (in megabytes) to
be allocated per driver in cluster mode. This is memory that accounts for
things like VM overheads, interned strings, other native overheads, etc.
This tends to grow with the container size (typically 6-10%).
spark.yarn.am.memoryOverheadAM memory * 0.07, with minimum of 384Same as
spark.yarn.driver.memoryOverhead, but for the Application Master in client
mode.

On Sun, Aug 2, 2015 at 12:02 PM, Paul Röwer <pa...@googlemail.com>
wrote:

> hi, what reason have this exception?
>
> Log Type: stderr
>
> Log Upload Time: Sun Aug 02 13:35:16 +0200 2015
>
> Log Length: 72197
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/yarn/nm/usercache/marcel/filecache/296/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 15/08/02 13:25:38 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
> 15/08/02 13:25:38 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438417242433_0206_000001
> 15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
> 15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
> 15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
> 15/08/02 13:25:38 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
> 15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization
> 15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
> 15/08/02 13:25:38 INFO spark.SparkContext: Running Spark version 1.3.0
> 15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
> 15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
> 15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
> 15/08/02 13:25:39 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/08/02 13:25:39 INFO Remoting: Starting remoting
> 15/08/02 13:25:39 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@node2-scads05:54443]
> 15/08/02 13:25:39 INFO util.Utils: Successfully started service 'sparkDriver' on port 54443.
> 15/08/02 13:25:39 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/08/02 13:25:39 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/08/02 13:25:39 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/blockmgr-74d8b61c-430c-4d02-bb56-f3fe85c8df1b
> 15/08/02 13:25:39 INFO storage.MemoryStore: MemoryStore started with capacity 245.7 MB
> 15/08/02 13:25:39 INFO spark.HttpFileServer: HTTP File server directory is /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/httpd-7bcf27d9-42ff-4ac3-9296-39afffd559d1
> 15/08/02 13:25:39 INFO spark.HttpServer: Starting HTTP Server
> 15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/08/02 13:25:39 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42691
> 15/08/02 13:25:39 INFO util.Utils: Successfully started service 'HTTP file server' on port 42691.
> 15/08/02 13:25:39 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/08/02 13:25:39 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/08/02 13:25:39 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:41412
> 15/08/02 13:25:39 INFO util.Utils: Successfully started service 'SparkUI' on port 41412.
> 15/08/02 13:25:39 INFO ui.SparkUI: Started SparkUI at http://node2-scads05:41412
> 15/08/02 13:25:39 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
> 15/08/02 13:25:39 INFO netty.NettyBlockTransferService: Server created on 50154
> 15/08/02 13:25:39 INFO storage.BlockManagerMaster: Trying to register BlockManager
> 15/08/02 13:25:39 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:50154 with 245.7 MB RAM, BlockManagerId(<driver>, node2-scads05, 50154)
> 15/08/02 13:25:39 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/08/02 13:25:39 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@node2-scads05:54443/user/YarnScheduler
> 15/08/02 13:25:39 INFO cluster.YarnClusterSchedulerBackend: ApplicationMaster registered as Actor[akka://sparkDriver/user/YarnAM#347751435]
> 15/08/02 13:25:39 INFO client.RMProxy: Connecting to ResourceManager at node1-scads02/139.18.13.42:8030
> 15/08/02 13:25:39 INFO yarn.YarnRMClient: Registering the ApplicationMaster
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:25:39 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
> 15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node3-scads06:8041
> 15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node2-scads05:8041
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000002 for on host node3-scads06
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000003 for on host node2-scads05
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:25:39 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
> 15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 1, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 2, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
> 15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
> 15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35454/user/Executor#-243069105] with ID 2
> 15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:34516 with 3.3 GB RAM, BlockManagerId(2, node2-scads05, 34516)
> 15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:58198/user/Executor#169360033] with ID 1
> 15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
> 15/08/02 13:25:41 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
> 15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58512 with 3.3 GB RAM, BlockManagerId(1, node3-scads06, 58512)
> 15/08/02 13:25:41 INFO storage.MemoryStore: ensureFreeSpace(115917) called with curMem=0, maxMem=257635123
> 15/08/02 13:25:41 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 113.2 KB, free 245.6 MB)
> 15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(26058) called with curMem=115917, maxMem=257635123
> 15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.4 KB, free 245.6 MB)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:50154 (size: 25.4 KB, free: 245.7 MB)
> 15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
> 15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 0 from textFile at SparkBenchmarkLauncher.java:71
> 15/08/02 13:25:42 INFO mapred.FileInputFormat: Total input paths to process : 2
> 15/08/02 13:25:42 INFO spark.SparkContext: Starting job: collect at SparkBenchmarkLauncher.java:246
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Registering RDD 4 (mapToPair at SparkBenchmarkLauncher.java:181)
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Got job 0 (collect at SparkBenchmarkLauncher.java:246) with 4 output partitions (allowLocal=false)
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Final stage: Stage 1(collect at SparkBenchmarkLauncher.java:246)
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181), which has no missing parents
> 15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(5640) called with curMem=141975, maxMem=257635123
> 15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.5 KB, free 245.6 MB)
> 15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(3310) called with curMem=147615, maxMem=257635123
> 15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.2 KB, free 245.6 MB)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:50154 (size: 3.2 KB, free: 245.7 MB)
> 15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
> 15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
> 15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181)
> 15/08/02 13:25:42 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 4 tasks
> 15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, node3-scads06, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, node2-scads05, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58512 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:34516 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:34516 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58512 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:25:44 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:34516 (size: 18.8 MB, free: 3.3 GB)
> 15/08/02 13:25:45 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58512 (size: 44.1 MB, free: 3.2 GB)
> 15/08/02 13:26:40 ERROR cluster.YarnClusterScheduler: Lost executor 1 on node3-scads06: remote Akka client disassociated
> 15/08/02 13:26:40 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:58198] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:26:40 INFO scheduler.TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
> 15/08/02 13:26:40 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, node3-scads06): ExecutorLostFailure (executor 1 lost)
> 15/08/02 13:26:40 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 1
> 15/08/02 13:26:40 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 0)
> 15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
> 15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(1, node3-scads06, 58512)
> 15/08/02 13:26:40 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
> 15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000003 (state: COMPLETE, exit status: -104)
> 15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000002 (state: COMPLETE, exit status: -104)
> 15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.0 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:26:44 ERROR cluster.YarnClusterScheduler: Lost executor 2 on node2-scads05: remote Akka client disassociated
> 15/08/02 13:26:44 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35454] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:26:44 INFO scheduler.TaskSetManager: Re-queueing tasks for 2 from TaskSet 0.0
> 15/08/02 13:26:44 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, node2-scads05): ExecutorLostFailure (executor 2 lost)
> 15/08/02 13:26:44 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 2
> 15/08/02 13:26:44 INFO scheduler.DAGScheduler: Executor lost: 2 (epoch 1)
> 15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster.
> 15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(2, node2-scads05, 34516)
> 15/08/02 13:26:44 INFO storage.BlockManagerMaster: Removed 2 successfully in removeExecutor
> 15/08/02 13:26:49 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000004 for on host node3-scads06
> 15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000005 for on host node2-scads05
> 15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:26:54 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
> 15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 4, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 3, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
> 15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35705/user/Executor#1553649062] with ID 4
> 15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 0.0 (TID 2, node2-scads05, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:57874/user/Executor#723963404] with ID 3
> 15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 3, node3-scads06, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40130 with 3.3 GB RAM, BlockManagerId(4, node2-scads05, 40130)
> 15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:42588 with 3.3 GB RAM, BlockManagerId(3, node3-scads06, 42588)
> 15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40130 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:42588 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:42588 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40130 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:26:58 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40130 (size: 18.8 MB, free: 3.3 GB)
> 15/08/02 13:26:59 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:42588 (size: 44.1 MB, free: 3.2 GB)
> 15/08/02 13:27:57 ERROR cluster.YarnClusterScheduler: Lost executor 4 on node2-scads05: remote Akka client disassociated
> 15/08/02 13:27:57 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35705] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:27:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 4 from TaskSet 0.0
> 15/08/02 13:27:57 WARN scheduler.TaskSetManager: Lost task 1.1 in stage 0.0 (TID 2, node2-scads05): ExecutorLostFailure (executor 4 lost)
> 15/08/02 13:27:57 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 4
> 15/08/02 13:27:57 INFO scheduler.DAGScheduler: Executor lost: 4 (epoch 2)
> 15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Trying to remove executor 4 from BlockManagerMaster.
> 15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(4, node2-scads05, 40130)
> 15/08/02 13:27:57 INFO storage.BlockManagerMaster: Removed 4 successfully in removeExecutor
> 15/08/02 13:27:59 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000005 (state: COMPLETE, exit status: -104)
> 15/08/02 13:27:59 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:28:01 ERROR cluster.YarnClusterScheduler: Lost executor 3 on node3-scads06: remote Akka client disassociated
> 15/08/02 13:28:01 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:57874] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:28:01 INFO scheduler.TaskSetManager: Re-queueing tasks for 3 from TaskSet 0.0
> 15/08/02 13:28:01 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 3, node3-scads06): ExecutorLostFailure (executor 3 lost)
> 15/08/02 13:28:01 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 3
> 15/08/02 13:28:01 INFO scheduler.DAGScheduler: Executor lost: 3 (epoch 3)
> 15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster.
> 15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(3, node3-scads06, 42588)
> 15/08/02 13:28:01 INFO storage.BlockManagerMaster: Removed 3 successfully in removeExecutor
> 15/08/02 13:28:04 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:28:04 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:28:04 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000004 (state: COMPLETE, exit status: -104)
> 15/08/02 13:28:04 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:28:09 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:28:09 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000006 for on host node3-scads06
> 15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:28:09 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
> 15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 5, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
> 15/08/02 13:28:11 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:34403/user/Executor#312841975] with ID 5
> 15/08/02 13:28:11 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, node3-scads06, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:28:11 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58302 with 3.3 GB RAM, BlockManagerId(5, node3-scads06, 58302)
> 15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58302 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58302 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:28:14 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58302 (size: 44.1 MB, free: 3.2 GB)
> 15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000007 for on host node2-scads05
> 15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:28:14 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
> 15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 6, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
> 15/08/02 13:28:16 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:44899/user/Executor#112632092] with ID 6
> 15/08/02 13:28:16 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, node2-scads05, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:28:16 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40856 with 3.3 GB RAM, BlockManagerId(6, node2-scads05, 40856)
> 15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40856 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40856 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:28:18 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40856 (size: 18.8 MB, free: 3.3 GB)
> 15/08/02 13:29:10 ERROR cluster.YarnClusterScheduler: Lost executor 5 on node3-scads06: remote Akka client disassociated
> 15/08/02 13:29:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:34403] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:29:10 INFO scheduler.TaskSetManager: Re-queueing tasks for 5 from TaskSet 0.0
> 15/08/02 13:29:10 WARN scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4, node3-scads06): ExecutorLostFailure (executor 5 lost)
> 15/08/02 13:29:10 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 5
> 15/08/02 13:29:10 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 4)
> 15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Trying to remove executor 5 from BlockManagerMaster.
> 15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(5, node3-scads06, 58302)
> 15/08/02 13:29:10 INFO storage.BlockManagerMaster: Removed 5 successfully in removeExecutor
> 15/08/02 13:29:14 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000006 (state: COMPLETE, exit status: -104)
> 15/08/02 13:29:14 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:29:18 ERROR cluster.YarnClusterScheduler: Lost executor 6 on node2-scads05: remote Akka client disassociated
> 15/08/02 13:29:18 INFO scheduler.TaskSetManager: Re-queueing tasks for 6 from TaskSet 0.0
> 15/08/02 13:29:18 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:44899] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:29:18 WARN scheduler.TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5, node2-scads05): ExecutorLostFailure (executor 6 lost)
> 15/08/02 13:29:18 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 6
> 15/08/02 13:29:18 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 5)
> 15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Trying to remove executor 6 from BlockManagerMaster.
> 15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(6, node2-scads05, 40856)
> 15/08/02 13:29:18 INFO storage.BlockManagerMaster: Removed 6 successfully in removeExecutor
> 15/08/02 13:29:19 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:29:19 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:29:19 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000007 (state: COMPLETE, exit status: -104)
> 15/08/02 13:29:19 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:29:24 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
> 15/08/02 13:29:24 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
> 15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000009 for on host node3-scads06
> 15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
> 15/08/02 13:29:24 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 7, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
> 15/08/02 13:29:26 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:49082/user/Executor#624674070] with ID 7
> 15/08/02 13:29:26 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 0.0 (TID 6, node3-scads06, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:29:26 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:36761 with 3.3 GB RAM, BlockManagerId(7, node3-scads06, 36761)
> 15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:36761 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:36761 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:29:28 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node3-scads06:36761 (size: 18.8 MB, free: 3.3 GB)
> 15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000010 for on host node2-scads05
> 15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
> 15/08/02 13:29:29 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
>
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
> 15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 8, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
> 15/08/02 13:29:31 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:36737/user/Executor#-1106619214] with ID 8
> 15/08/02 13:29:31 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 7, node2-scads05, NODE_LOCAL, 1316 bytes)
> 15/08/02 13:29:31 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:44866 with 3.3 GB RAM, BlockManagerId(8, node2-scads05, 44866)
> 15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:44866 (size: 3.2 KB, free: 3.3 GB)
> 15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:44866 (size: 25.4 KB, free: 3.3 GB)
> 15/08/02 13:29:34 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node2-scads05:44866 (size: 44.1 MB, free: 3.2 GB)
> 15/08/02 13:30:28 ERROR cluster.YarnClusterScheduler: Lost executor 7 on node3-scads06: remote Akka client disassociated
> 15/08/02 13:30:28 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:49082] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/08/02 13:30:28 INFO scheduler.TaskSetManager: Re-queueing tasks for 7 from TaskSet 0.0
> 15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
> 15/08/02 13:30:28 ERROR scheduler.TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job
> 15/08/02 13:30:28 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 7
> 15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Cancelling stage 0
> 15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Job 0 failed: collect at SparkBenchmarkLauncher.java:246, took 286.500864 s
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
> Driver stacktrace:
> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 	at scala.Option.foreach(Option.scala:236)
> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Executor lost: 7 (epoch 6)
> 15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Trying to remove executor 7 from BlockManagerMaster.
> 15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(7, node3-scads06, 36761)
> 15/08/02 13:30:28 INFO storage.BlockManagerMaster: Removed 7 successfully in removeExecutor
> 15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 7, node2-scads05): TaskKilled (killed intentionally)
> 15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
> 15/08/02 13:30:28 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
> 15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
> 15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
> 15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
> 15/08/02 13:30:28 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
> 15/08/02 13:30:28 INFO spark.SparkContext: Starting job: saveAsTextFile at SparkBenchmarkLauncher.java:293
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Got job 1 (saveAsTextFile at SparkBenchmarkLauncher.java:293) with 4 output partitions (allowLocal=false)
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Final stage: Stage 2(saveAsTextFile at SparkBenchmarkLauncher.java:293)
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Parents of final stage: List()
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Missing parents: List()
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293), which has no missing parents
> 15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(121336) called with curMem=150925, maxMem=257635123
> 15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 118.5 KB, free 245.4 MB)
> 15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(57208) called with curMem=272261, maxMem=257635123
> 15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 55.9 KB, free 245.4 MB)
> 15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:50154 (size: 55.9 KB, free: 245.6 MB)
> 15/08/02 13:30:28 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
> 15/08/02 13:30:28 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
> 15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293)
> 15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Adding task set 2.0 with 4 tasks
> 15/08/02 13:30:28 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, node2-scads05, PROCESS_LOCAL, 1327 bytes)
> 15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:44866 (size: 55.9 KB, free: 3.2 GB)
> 15/08/02 13:30:29 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, node2-scads05, NODE_LOCAL, 1327 bytes)
> 15/08/02 13:30:29 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 690 ms on node2-scads05 (1/4)
> 15/08/02 13:30:29 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000009 (state: COMPLETE, exit status: -104)
> 15/08/02 13:30:29 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
> 15/08/02 13:30:30 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:44866 (size: 18.8 MB, free: 3.2 GB)
> 15/08/02 13:30:30 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, node2-scads05, NODE_LOCAL, 1327 bytes)
> 15/08/02 13:30:30 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 926 ms on node2-scads05 (2/4)
> 15/08/02 13:30:32 INFO storage.BlockManagerInfo: Added rdd_3_2 in memory on node2-scads05:44866 (size: 43.8 MB, free: 3.2 GB)
> 15/08/02 13:30:32 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, node2-scads05, NODE_LOCAL, 1327 bytes)
> 15/08/02 13:30:32 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 1835 ms on node2-scads05 (3/4)
> 15/08/02 13:30:33 INFO storage.BlockManagerInfo: Added rdd_3_3 in memory on node2-scads05:44866 (size: 19.2 MB, free: 3.2 GB)
> 15/08/02 13:30:33 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 885 ms on node2-scads05 (4/4)
> 15/08/02 13:30:33 INFO cluster.YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool
> 15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stage 2 (saveAsTextFile at SparkBenchmarkLauncher.java:293) finished in 4.336 s
> 15/08/02 13:30:33 INFO scheduler.DAGScheduler: Job 1 finished: saveAsTextFile at SparkBenchmarkLauncher.java:293, took 4.379196 s
> 15/08/02 13:30:33 ERROR yarn.ApplicationMaster: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
> 	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
>
> java.io.FileNotFoundException: File does not exist: /user/marcel/outputs/output_spark/log0
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
> 	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
>
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> 	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> 	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
> 	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
> 	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
> 	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
> 	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
> 	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
> 	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
> 	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
> 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
> 	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.analyseIterationLogandDeleteThem(SparkBenchmarkLauncher.java:147)
> 	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.main(SparkBenchmarkLauncher.java:132)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/marcel/outputs/output_spark/log0
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
> 	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
>
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> 	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> 	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
> 	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1142)
> 	... 18 more
> 15/08/02 13:30:33 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> 	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
> 	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
> )
> 15/08/02 13:30:33 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown hook
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
> 15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
> 15/08/02 13:30:33 INFO ui.SparkUI: Stopped Spark web UI at http://node2-scads05:41412
> 15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stopping DAGScheduler
> 15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
> 15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
> 15/08/02 13:30:33 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
> 15/08/02 13:30:33 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
> 15/08/02 13:30:33 INFO storage.MemoryStore: MemoryStore cleared
> 15/08/02 13:30:33 INFO storage.BlockManager: BlockManager stopped
> 15/08/02 13:30:33 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
> 15/08/02 13:30:33 INFO spark.SparkContext: Successfully stopped SparkContext
>
>
>
> Am 02.08.2015 um 16:23 schrieb Connor Zanin:
>
> I agree with Ted. Could you please post the log file?
> On Aug 2, 2015 10:13 AM, "Ted Yu" <yu...@gmail.com> wrote:
>
>> Can you provide some more detai:
>>
>> release of Spark you're using
>> were you running in standalone or YARN cluster mode
>> have you checked driver log ?
>>
>> Cheers
>>
>> On Sun, Aug 2, 2015 at 7:04 AM, Pa Rö <pa...@googlemail.com>
>> wrote:
>>
>>> hi community,
>>>
>>> i have run my k-means spark application on 1million data points. the
>>> program works, but no output in the hdfs is generated. when it runs on
>>> 10.000 points, a output is written.
>>>
>>> maybe someone has an idea?
>>>
>>> best regards,
>>> paul
>>>
>>
>>
>

Re: spark no output

Posted by Paul Röwer <pa...@googlemail.com>.

hi, what reason have this exception?

Log Type: stderr

Log Upload Time: Sun Aug 02 13:35:16 +0200 2015

Log Length: 72197

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/yarn/nm/usercache/marcel/filecache/296/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/08/02 13:25:38 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438417242433_0206_000001
15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
15/08/02 13:25:38 INFO spark.SparkContext: Running Spark version 1.3.0
15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
15/08/02 13:25:39 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/08/02 13:25:39 INFO Remoting: Starting remoting
15/08/02 13:25:39 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@node2-scads05:54443]
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'sparkDriver' on port 54443.
15/08/02 13:25:39 INFO spark.SparkEnv: Registering MapOutputTracker
15/08/02 13:25:39 INFO spark.SparkEnv: Registering BlockManagerMaster
15/08/02 13:25:39 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/blockmgr-74d8b61c-430c-4d02-bb56-f3fe85c8df1b
15/08/02 13:25:39 INFO storage.MemoryStore: MemoryStore started with capacity 245.7 MB
15/08/02 13:25:39 INFO spark.HttpFileServer: HTTP File server directory is /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/httpd-7bcf27d9-42ff-4ac3-9296-39afffd559d1
15/08/02 13:25:39 INFO spark.HttpServer: Starting HTTP Server
15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/02 13:25:39 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42691
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'HTTP file server' on port 42691.
15/08/02 13:25:39 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/08/02 13:25:39 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/02 13:25:39 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:41412
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'SparkUI' on port 41412.
15/08/02 13:25:39 INFO ui.SparkUI: Started SparkUI at http://node2-scads05:41412
15/08/02 13:25:39 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
15/08/02 13:25:39 INFO netty.NettyBlockTransferService: Server created on 50154
15/08/02 13:25:39 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/08/02 13:25:39 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:50154 with 245.7 MB RAM, BlockManagerId(<driver>, node2-scads05, 50154)
15/08/02 13:25:39 INFO storage.BlockManagerMaster: Registered BlockManager
15/08/02 13:25:39 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@node2-scads05:54443/user/YarnScheduler
15/08/02 13:25:39 INFO cluster.YarnClusterSchedulerBackend: ApplicationMaster registered as Actor[akka://sparkDriver/user/YarnAM#347751435]
15/08/02 13:25:39 INFO client.RMProxy: Connecting to ResourceManager at node1-scads02/139.18.13.42:8030
15/08/02 13:25:39 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/08/02 13:25:39 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:25:39 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node3-scads06:8041
15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node2-scads05:8041
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000002 for on host node3-scads06
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000003 for on host node2-scads05
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:25:39 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 1, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 2, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35454/user/Executor#-243069105] with ID 2
15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:34516 with 3.3 GB RAM, BlockManagerId(2, node2-scads05, 34516)
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:58198/user/Executor#169360033] with ID 1
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
15/08/02 13:25:41 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58512 with 3.3 GB RAM, BlockManagerId(1, node3-scads06, 58512)
15/08/02 13:25:41 INFO storage.MemoryStore: ensureFreeSpace(115917) called with curMem=0, maxMem=257635123
15/08/02 13:25:41 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 113.2 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(26058) called with curMem=115917, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.4 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:50154 (size: 25.4 KB, free: 245.7 MB)
15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 0 from textFile at SparkBenchmarkLauncher.java:71
15/08/02 13:25:42 INFO mapred.FileInputFormat: Total input paths to process : 2
15/08/02 13:25:42 INFO spark.SparkContext: Starting job: collect at SparkBenchmarkLauncher.java:246
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Registering RDD 4 (mapToPair at SparkBenchmarkLauncher.java:181)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Got job 0 (collect at SparkBenchmarkLauncher.java:246) with 4 output partitions (allowLocal=false)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Final stage: Stage 1(collect at SparkBenchmarkLauncher.java:246)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181), which has no missing parents
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(5640) called with curMem=141975, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.5 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(3310) called with curMem=147615, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.2 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:50154 (size: 3.2 KB, free: 245.7 MB)
15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181)
15/08/02 13:25:42 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 4 tasks
15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58512 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:34516 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:34516 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58512 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:25:44 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:34516 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:25:45 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58512 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:26:40 ERROR cluster.YarnClusterScheduler: Lost executor 1 on node3-scads06: remote Akka client disassociated
15/08/02 13:26:40 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:58198] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:26:40 INFO scheduler.TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
15/08/02 13:26:40 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, node3-scads06): ExecutorLostFailure (executor 1 lost)
15/08/02 13:26:40 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 1
15/08/02 13:26:40 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 0)
15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(1, node3-scads06, 58512)
15/08/02 13:26:40 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000003 (state: COMPLETE, exit status: -104)
15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000002 (state: COMPLETE, exit status: -104)
15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.0 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:26:44 ERROR cluster.YarnClusterScheduler: Lost executor 2 on node2-scads05: remote Akka client disassociated
15/08/02 13:26:44 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35454] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:26:44 INFO scheduler.TaskSetManager: Re-queueing tasks for 2 from TaskSet 0.0
15/08/02 13:26:44 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, node2-scads05): ExecutorLostFailure (executor 2 lost)
15/08/02 13:26:44 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 2
15/08/02 13:26:44 INFO scheduler.DAGScheduler: Executor lost: 2 (epoch 1)
15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster.
15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(2, node2-scads05, 34516)
15/08/02 13:26:44 INFO storage.BlockManagerMaster: Removed 2 successfully in removeExecutor
15/08/02 13:26:49 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000004 for on host node3-scads06
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000005 for on host node2-scads05
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:26:54 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 4, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 3, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35705/user/Executor#1553649062] with ID 4
15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 0.0 (TID 2, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:57874/user/Executor#723963404] with ID 3
15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 3, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40130 with 3.3 GB RAM, BlockManagerId(4, node2-scads05, 40130)
15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:42588 with 3.3 GB RAM, BlockManagerId(3, node3-scads06, 42588)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40130 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:42588 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:42588 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40130 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:26:58 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40130 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:26:59 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:42588 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:27:57 ERROR cluster.YarnClusterScheduler: Lost executor 4 on node2-scads05: remote Akka client disassociated
15/08/02 13:27:57 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35705] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:27:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 4 from TaskSet 0.0
15/08/02 13:27:57 WARN scheduler.TaskSetManager: Lost task 1.1 in stage 0.0 (TID 2, node2-scads05): ExecutorLostFailure (executor 4 lost)
15/08/02 13:27:57 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 4
15/08/02 13:27:57 INFO scheduler.DAGScheduler: Executor lost: 4 (epoch 2)
15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Trying to remove executor 4 from BlockManagerMaster.
15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(4, node2-scads05, 40130)
15/08/02 13:27:57 INFO storage.BlockManagerMaster: Removed 4 successfully in removeExecutor
15/08/02 13:27:59 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000005 (state: COMPLETE, exit status: -104)
15/08/02 13:27:59 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:28:01 ERROR cluster.YarnClusterScheduler: Lost executor 3 on node3-scads06: remote Akka client disassociated
15/08/02 13:28:01 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:57874] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:28:01 INFO scheduler.TaskSetManager: Re-queueing tasks for 3 from TaskSet 0.0
15/08/02 13:28:01 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 3, node3-scads06): ExecutorLostFailure (executor 3 lost)
15/08/02 13:28:01 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 3
15/08/02 13:28:01 INFO scheduler.DAGScheduler: Executor lost: 3 (epoch 3)
15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster.
15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(3, node3-scads06, 42588)
15/08/02 13:28:01 INFO storage.BlockManagerMaster: Removed 3 successfully in removeExecutor
15/08/02 13:28:04 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:28:04 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:28:04 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000004 (state: COMPLETE, exit status: -104)
15/08/02 13:28:04 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:28:09 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:28:09 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000006 for on host node3-scads06
15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:28:09 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 5, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:28:11 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:34403/user/Executor#312841975] with ID 5
15/08/02 13:28:11 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:28:11 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58302 with 3.3 GB RAM, BlockManagerId(5, node3-scads06, 58302)
15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58302 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58302 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:28:14 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58302 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000007 for on host node2-scads05
15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:28:14 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 6, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:28:16 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:44899/user/Executor#112632092] with ID 6
15/08/02 13:28:16 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:28:16 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40856 with 3.3 GB RAM, BlockManagerId(6, node2-scads05, 40856)
15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40856 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40856 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:28:18 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40856 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:29:10 ERROR cluster.YarnClusterScheduler: Lost executor 5 on node3-scads06: remote Akka client disassociated
15/08/02 13:29:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:34403] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:29:10 INFO scheduler.TaskSetManager: Re-queueing tasks for 5 from TaskSet 0.0
15/08/02 13:29:10 WARN scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4, node3-scads06): ExecutorLostFailure (executor 5 lost)
15/08/02 13:29:10 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 5
15/08/02 13:29:10 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 4)
15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Trying to remove executor 5 from BlockManagerMaster.
15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(5, node3-scads06, 58302)
15/08/02 13:29:10 INFO storage.BlockManagerMaster: Removed 5 successfully in removeExecutor
15/08/02 13:29:14 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000006 (state: COMPLETE, exit status: -104)
15/08/02 13:29:14 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:29:18 ERROR cluster.YarnClusterScheduler: Lost executor 6 on node2-scads05: remote Akka client disassociated
15/08/02 13:29:18 INFO scheduler.TaskSetManager: Re-queueing tasks for 6 from TaskSet 0.0
15/08/02 13:29:18 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:44899] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:29:18 WARN scheduler.TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5, node2-scads05): ExecutorLostFailure (executor 6 lost)
15/08/02 13:29:18 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 6
15/08/02 13:29:18 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 5)
15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Trying to remove executor 6 from BlockManagerMaster.
15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(6, node2-scads05, 40856)
15/08/02 13:29:18 INFO storage.BlockManagerMaster: Removed 6 successfully in removeExecutor
15/08/02 13:29:19 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:29:19 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:29:19 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000007 (state: COMPLETE, exit status: -104)
15/08/02 13:29:19 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:29:24 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:29:24 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000009 for on host node3-scads06
15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:29:24 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 7, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:29:26 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:49082/user/Executor#624674070] with ID 7
15/08/02 13:29:26 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 0.0 (TID 6, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:29:26 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:36761 with 3.3 GB RAM, BlockManagerId(7, node3-scads06, 36761)
15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:36761 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:36761 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:29:28 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node3-scads06:36761 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000010 for on host node2-scads05
15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:29:29 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 8, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:29:31 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:36737/user/Executor#-1106619214] with ID 8
15/08/02 13:29:31 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 7, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:29:31 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:44866 with 3.3 GB RAM, BlockManagerId(8, node2-scads05, 44866)
15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:44866 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:44866 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:29:34 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node2-scads05:44866 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:30:28 ERROR cluster.YarnClusterScheduler: Lost executor 7 on node3-scads06: remote Akka client disassociated
15/08/02 13:30:28 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:49082] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:30:28 INFO scheduler.TaskSetManager: Re-queueing tasks for 7 from TaskSet 0.0
15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
15/08/02 13:30:28 ERROR scheduler.TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job
15/08/02 13:30:28 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 7
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Cancelling stage 0
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Job 0 failed: collect at SparkBenchmarkLauncher.java:246, took 286.500864 s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Executor lost: 7 (epoch 6)
15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Trying to remove executor 7 from BlockManagerMaster.
15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(7, node3-scads06, 36761)
15/08/02 13:30:28 INFO storage.BlockManagerMaster: Removed 7 successfully in removeExecutor
15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 7, node2-scads05): TaskKilled (killed intentionally)
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/08/02 13:30:28 INFO spark.SparkContext: Starting job: saveAsTextFile at SparkBenchmarkLauncher.java:293
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Got job 1 (saveAsTextFile at SparkBenchmarkLauncher.java:293) with 4 output partitions (allowLocal=false)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Final stage: Stage 2(saveAsTextFile at SparkBenchmarkLauncher.java:293)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Missing parents: List()
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293), which has no missing parents
15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(121336) called with curMem=150925, maxMem=257635123
15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 118.5 KB, free 245.4 MB)
15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(57208) called with curMem=272261, maxMem=257635123
15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 55.9 KB, free 245.4 MB)
15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:50154 (size: 55.9 KB, free: 245.6 MB)
15/08/02 13:30:28 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
15/08/02 13:30:28 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293)
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Adding task set 2.0 with 4 tasks
15/08/02 13:30:28 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, node2-scads05, PROCESS_LOCAL, 1327 bytes)
15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:44866 (size: 55.9 KB, free: 3.2 GB)
15/08/02 13:30:29 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:29 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 690 ms on node2-scads05 (1/4)
15/08/02 13:30:29 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000009 (state: COMPLETE, exit status: -104)
15/08/02 13:30:29 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:30:30 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:44866 (size: 18.8 MB, free: 3.2 GB)
15/08/02 13:30:30 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:30 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 926 ms on node2-scads05 (2/4)
15/08/02 13:30:32 INFO storage.BlockManagerInfo: Added rdd_3_2 in memory on node2-scads05:44866 (size: 43.8 MB, free: 3.2 GB)
15/08/02 13:30:32 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:32 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 1835 ms on node2-scads05 (3/4)
15/08/02 13:30:33 INFO storage.BlockManagerInfo: Added rdd_3_3 in memory on node2-scads05:44866 (size: 19.2 MB, free: 3.2 GB)
15/08/02 13:30:33 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 885 ms on node2-scads05 (4/4)
15/08/02 13:30:33 INFO cluster.YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stage 2 (saveAsTextFile at SparkBenchmarkLauncher.java:293) finished in 4.336 s
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Job 1 finished: saveAsTextFile at SparkBenchmarkLauncher.java:293, took 4.379196 s
15/08/02 13:30:33 ERROR yarn.ApplicationMaster: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

java.io.FileNotFoundException: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.analyseIterationLogandDeleteThem(SparkBenchmarkLauncher.java:147)
	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.main(SparkBenchmarkLauncher.java:132)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

	at org.apache.hadoop.ipc.Client.call(Client.java:1410)
	at org.apache.hadoop.ipc.Client.call(Client.java:1363)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1142)
	... 18 more
15/08/02 13:30:33 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
)
15/08/02 13:30:33 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown hook
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
15/08/02 13:30:33 INFO ui.SparkUI: Stopped Spark web UI at http://node2-scads05:41412
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
15/08/02 13:30:33 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
15/08/02 13:30:33 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
15/08/02 13:30:33 INFO storage.MemoryStore: MemoryStore cleared
15/08/02 13:30:33 INFO storage.BlockManager: BlockManager stopped
15/08/02 13:30:33 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
15/08/02 13:30:33 INFO spark.SparkContext: Successfully stopped SparkContext



Am 02.08.2015 um 16:23 schrieb Connor Zanin:
>
> I agree with Ted. Could you please post the log file?
>
> On Aug 2, 2015 10:13 AM, "Ted Yu" <yuzhihong@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     Can you provide some more detai:
>
>     release of Spark you're using
>     were you running in standalone or YARN cluster mode
>     have you checked driver log ?
>
>     Cheers
>
>     On Sun, Aug 2, 2015 at 7:04 AM, Pa Rö
>     <paul.roewer1990@googlemail.com
>     <ma...@googlemail.com>> wrote:
>
>         hi community,
>
>         i have run my k-means spark application on 1million data
>         points. the program works, but no output in the hdfs is
>         generated. when it runs on 10.000 points, a output is written.
>
>         maybe someone has an idea?
>
>         best regards,
>         paul
>
>

Re: spark no output

Posted by Connor Zanin <cn...@udel.edu>.

I agree with Ted. Could you please post the log file?
On Aug 2, 2015 10:13 AM, "Ted Yu" <yu...@gmail.com> wrote:

> Can you provide some more detai:
>
> release of Spark you're using
> were you running in standalone or YARN cluster mode
> have you checked driver log ?
>
> Cheers
>
> On Sun, Aug 2, 2015 at 7:04 AM, Pa Rö <pa...@googlemail.com>
> wrote:
>
>> hi community,
>>
>> i have run my k-means spark application on 1million data points. the
>> program works, but no output in the hdfs is generated. when it runs on
>> 10.000 points, a output is written.
>>
>> maybe someone has an idea?
>>
>> best regards,
>> paul
>>
>
>

Re: spark no output

Posted by Ted Yu <yu...@gmail.com>.

Can you provide some more detai:

release of Spark you're using
were you running in standalone or YARN cluster mode
have you checked driver log ?

Cheers

On Sun, Aug 2, 2015 at 7:04 AM, Pa Rö <pa...@googlemail.com>
wrote:

> hi community,
>
> i have run my k-means spark application on 1million data points. the
> program works, but no output in the hdfs is generated. when it runs on
> 10.000 points, a output is written.
>
> maybe someone has an idea?
>
> best regards,
> paul
>