You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeetendra Gangele <ga...@gmail.com> on 2015/07/30 08:51:01 UTC
Spark on YARN
I am running below command this is default spark PI program but this is not
running all the log are going in stderr but at the terminal job is
succeeding .I guess there are con issue job it not at all launching
/bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10
Complete log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal
handlers for [TERM, HUP, INT]
15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId:
appattempt_1438090734187_0010_000001
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user
application in a separate Thread
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark
context initialization
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark
context initialization ...
15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
15/07/30 12:13:33 WARN spark.SparkConf:
SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with conf/spark-defaults.conf to set defaults for an
application
- ./spark-submit with --driver-java-options to set -X options for a driver
- spark.executor.extraJavaOptions to set -X options for executors
- SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons
(master or worker)
15/07/30 12:13:33 WARN spark.SparkConf: Setting
'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/07/30 12:13:33 WARN spark.SparkConf: Setting
'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/07/30 12:13:33 INFO Remoting: Starting remoting
15/07/30 12:13:34 INFO Remoting: Remoting started; listening on
addresses :[akka.tcp://sparkDriver@10.21.1.77:53411]
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'sparkDriver' on port 53411.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local
directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with
capacity 246.0 MB
15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server
directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:52507
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP
file server' on port 52507.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:59596
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'SparkUI' on port 59596.
15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
15/07/30 12:13:34 INFO util.Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port
43354.
15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering
block manager 10.21.1.77:43354 with 246.0 MB RAM,
BlockManagerId(driver, 10.21.1.77, 43354)
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
15/07/30 12:13:34 INFO
cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster
registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager
at hadoop-1/10.21.1.77:8030
15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor
containers, each with 1 cores and 1408 MB memory including 384 MB
overhead
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress
reporter thread - sleep time : 5000
15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status:
SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final
status was reported.)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering
ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called
before final status was reported.)
15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to
be successfully unregistered.
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging
directory .sparkStaging/application_1438090734187_0010
15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Deleting directory
/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
15/07/30 12:13:35 INFO util.Utils: Deleting directory
/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
Re: Spark on YARN
Posted by Jeetendra Gangele <ga...@gmail.com>.
Thanks for information this fixed the issue. Issue was in spark-master
memory when I specify manually 1G for master. it start working
On 30 July 2015 at 14:26, Shao, Saisai <sa...@intel.com> wrote:
> You’d better also check the log of nodemanager, sometimes because your
> memory usage exceeds the limit of Yarn container’s configuration.
>
>
>
> I’ve met similar problem before, here is the warning log in nodemanager:
>
>
>
> 2015-07-07 17:06:07,141 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Container [pid=17385,containerID=container_1436259427993_0001_02_000001] is
> running beyond virtual memory limits. Current usage: 318.1 MB of 1 GB
> physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing
> container.
>
>
>
> The default pmem-vmem ratio is 2.1, but seems executor requires more vmem
> when started, so nodemanager will kill it. If you met similar problem, you
> could increase this configuration “yarn.nodemanager.vmem-pmem-ratio”.
>
>
>
> Thanks
>
> Jerry
>
>
>
> *From:* Jeff Zhang [mailto:zjffdu@gmail.com]
> *Sent:* Thursday, July 30, 2015 4:36 PM
> *To:* Jeetendra Gangele
> *Cc:* user
> *Subject:* Re: Spark on YARN
>
>
>
> >> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15:
> SIGTERM
>
>
>
> AM is killed somehow, may due to preemption. Does it always happen ?
> Resource manager log would be helpful.
>
>
>
>
>
>
>
> On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
> I can't see the application logs here. All the logs are going into
> stderr. can anybody help here?
>
>
>
> On 30 July 2015 at 12:21, Jeetendra Gangele <ga...@gmail.com> wrote:
>
> I am running below command this is default spark PI program but this is
> not running all the log are going in stderr but at the terminal job is
> succeeding .I guess there are con issue job it not at all launching
>
>
>
> /bin/spark-submit --class org.apache.spark.examples.SparkPi --master
> yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10
>
>
>
>
>
> Complete log
>
>
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
>
> 15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438090734187_0010_000001
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
>
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
>
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization
>
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
>
> 15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
>
> 15/07/30 12:13:33 WARN spark.SparkConf:
>
> SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with conf/spark-defaults.conf to set defaults for an application
>
> - ./spark-submit with --driver-java-options to set -X options for a driver
>
> - spark.executor.extraJavaOptions to set -X options for executors
>
> - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker)
>
>
>
> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
>
> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
>
> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
>
> 15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
>
> 15/07/30 12:13:33 INFO Remoting: Starting remoting
>
> 15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.21.1.77:53411]
>
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53411.
>
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
>
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
>
> 15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
>
> 15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity 246.0 MB
>
> 15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
>
> 15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
>
> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52507
>
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file server' on port 52507.
>
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>
> 15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>
> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
>
> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:59596
>
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on port 59596.
>
> 15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
>
> 15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
>
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354.
>
> 15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
>
> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
>
> 15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.21.1.77:43354 with 246.0 MB RAM, BlockManagerId(driver, 10.21.1.77, 43354)
>
> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
>
> 15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
>
> 15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/10.21.1.77:8030
>
> 15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
>
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
>
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
>
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
>
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
>
> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
>
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
>
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
>
> 15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
>
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1438090734187_0010
>
> 15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
>
> 15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
>
> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
>
> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
>
>
>
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
RE: Spark on YARN
Posted by "Shao, Saisai" <sa...@intel.com>.
You’d better also check the log of nodemanager, sometimes because your memory usage exceeds the limit of Yarn container’s configuration.
I’ve met similar problem before, here is the warning log in nodemanager:
2015-07-07 17:06:07,141 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=17385,containerID=container_1436259427993_0001_02_000001] is running beyond virtual memory limits. Current usage: 318.1 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
The default pmem-vmem ratio is 2.1, but seems executor requires more vmem when started, so nodemanager will kill it. If you met similar problem, you could increase this configuration “yarn.nodemanager.vmem-pmem-ratio”.
Thanks
Jerry
From: Jeff Zhang [mailto:zjffdu@gmail.com]
Sent: Thursday, July 30, 2015 4:36 PM
To: Jeetendra Gangele
Cc: user
Subject: Re: Spark on YARN
>> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
AM is killed somehow, may due to preemption. Does it always happen ? Resource manager log would be helpful.
On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele <ga...@gmail.com>> wrote:
I can't see the application logs here. All the logs are going into stderr. can anybody help here?
On 30 July 2015 at 12:21, Jeetendra Gangele <ga...@gmail.com>> wrote:
I am running below command this is default spark PI program but this is not running all the log are going in stderr but at the terminal job is succeeding .I guess there are con issue job it not at all launching
/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10
Complete log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438090734187_0010_000001
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization
15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
15/07/30 12:13:33 WARN spark.SparkConf:
SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with conf/spark-defaults.conf to set defaults for an application
- ./spark-submit with --driver-java-options to set -X options for a driver
- spark.executor.extraJavaOptions to set -X options for executors
- SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker)
15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/07/30 12:13:33 INFO Remoting: Starting remoting
15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.21.1.77:53411<http://sparkDriver@10.21.1.77:53411>]
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53411.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity 246.0 MB
15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52507<http://SocketConnector@0.0.0.0:52507>
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file server' on port 52507.
15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/07/30 12:13:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:59596<http://SelectChannelConnector@0.0.0.0:59596>
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on port 59596.
15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
15/07/30 12:13:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354.
15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.21.1.77:43354<http://10.21.1.77:43354> with 246.0 MB RAM, BlockManagerId(driver, 10.21.1.77, 43354)
15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/10.21.1.77:8030<http://10.21.1.77:8030>
15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1438090734187_0010
15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
--
Best Regards
Jeff Zhang
Re: Spark on YARN
Posted by Jeff Zhang <zj...@gmail.com>.
>> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15:
SIGTERM
AM is killed somehow, may due to preemption. Does it always happen ?
Resource manager log would be helpful.
On Thu, Jul 30, 2015 at 4:17 PM, Jeetendra Gangele <ga...@gmail.com>
wrote:
> I can't see the application logs here. All the logs are going into stderr.
> can anybody help here?
>
> On 30 July 2015 at 12:21, Jeetendra Gangele <ga...@gmail.com> wrote:
>
>> I am running below command this is default spark PI program but this is
>> not running all the log are going in stderr but at the terminal job is
>> succeeding .I guess there are con issue job it not at all launching
>>
>> /bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>> yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10
>>
>>
>> Complete log
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> 15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
>> 15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438090734187_0010_000001
>> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
>> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
>> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
>> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
>> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization
>> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
>> 15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
>> 15/07/30 12:13:33 WARN spark.SparkConf:
>> SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
>> This is deprecated in Spark 1.0+.
>>
>> Please instead use:
>> - ./spark-submit with conf/spark-defaults.conf to set defaults for an application
>> - ./spark-submit with --driver-java-options to set -X options for a driver
>> - spark.executor.extraJavaOptions to set -X options for executors
>> - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker)
>>
>> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
>> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
>> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
>> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
>> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
>> 15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> 15/07/30 12:13:33 INFO Remoting: Starting remoting
>> 15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.21.1.77:53411]
>> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53411.
>> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
>> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
>> 15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity 246.0 MB
>> 15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
>> 15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
>> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52507
>> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file server' on port 52507.
>> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>> 15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:59596
>> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on port 59596.
>> 15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
>> 15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
>> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354.
>> 15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
>> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
>> 15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.21.1.77:43354 with 246.0 MB RAM, BlockManagerId(driver, 10.21.1.77, 43354)
>> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
>> 15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
>> 15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/10.21.1.77:8030
>> 15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
>> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
>> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
>> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
>> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
>> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
>> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
>> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
>> 15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
>> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1438090734187_0010
>> 15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
>> 15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
>> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
>> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
>>
>>
>
>
--
Best Regards
Jeff Zhang
Re: Spark on YARN
Posted by Jeetendra Gangele <ga...@gmail.com>.
I can't see the application logs here. All the logs are going into stderr.
can anybody help here?
On 30 July 2015 at 12:21, Jeetendra Gangele <ga...@gmail.com> wrote:
> I am running below command this is default spark PI program but this is
> not running all the log are going in stderr but at the terminal job is
> succeeding .I guess there are con issue job it not at all launching
>
> /bin/spark-submit --class org.apache.spark.examples.SparkPi --master
> yarn-cluster lib/spark-examples-1.4.1-hadoop2.6.0.jar 10
>
>
> Complete log
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/23/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 15/07/30 12:13:31 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
> 15/07/30 12:13:32 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438090734187_0010_000001
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization
> 15/07/30 12:13:33 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
> 15/07/30 12:13:33 INFO spark.SparkContext: Running Spark version 1.4.1
> 15/07/30 12:13:33 WARN spark.SparkConf:
> SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
> This is deprecated in Spark 1.0+.
>
> Please instead use:
> - ./spark-submit with conf/spark-defaults.conf to set defaults for an application
> - ./spark-submit with --driver-java-options to set -X options for a driver
> - spark.executor.extraJavaOptions to set -X options for executors
> - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or worker)
>
> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
> 15/07/30 12:13:33 WARN spark.SparkConf: Setting 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a work-around.
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing view acls to: hadoop
> 15/07/30 12:13:33 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 15/07/30 12:13:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
> 15/07/30 12:13:33 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/07/30 12:13:33 INFO Remoting: Starting remoting
> 15/07/30 12:13:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.21.1.77:53411]
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'sparkDriver' on port 53411.
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/07/30 12:13:34 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/blockmgr-2166bbd9-b1ed-41d1-bc95-92c6a7fbd36f
> 15/07/30 12:13:34 INFO storage.MemoryStore: MemoryStore started with capacity 246.0 MB
> 15/07/30 12:13:34 INFO spark.HttpFileServer: HTTP File server directory is /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
> 15/07/30 12:13:34 INFO spark.HttpServer: Starting HTTP Server
> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52507
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'HTTP file server' on port 52507.
> 15/07/30 12:13:34 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/07/30 12:13:34 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/07/30 12:13:34 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/07/30 12:13:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:59596
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'SparkUI' on port 59596.
> 15/07/30 12:13:34 INFO ui.SparkUI: Started SparkUI at http://10.21.1.77:59596
> 15/07/30 12:13:34 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
> 15/07/30 12:13:34 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43354.
> 15/07/30 12:13:34 INFO netty.NettyBlockTransferService: Server created on 43354
> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Trying to register BlockManager
> 15/07/30 12:13:34 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.21.1.77:43354 with 246.0 MB RAM, BlockManagerId(driver, 10.21.1.77, 43354)
> 15/07/30 12:13:34 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/07/30 12:13:34 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#-603094240])
> 15/07/30 12:13:34 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/10.21.1.77:8030
> 15/07/30 12:13:34 INFO yarn.YarnRMClient: Registering the ApplicationMaster
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
> 15/07/30 12:13:34 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
> 15/07/30 12:13:35 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
> 15/07/30 12:13:35 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
> 15/07/30 12:13:35 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1438090734187_0010
> 15/07/30 12:13:35 INFO storage.DiskBlockManager: Shutdown hook called
> 15/07/30 12:13:35 INFO util.Utils: Shutdown hook called
> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/userFiles-337c9be5-569f-43ff-ba1f-ec24daab9ea5
> 15/07/30 12:13:35 INFO util.Utils: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1438090734187_0010/httpd-d1232310-5aa1-44e7-a99a-cc2ae614f89c
>
>