You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by innowireless TaeYun Kim <ta...@innowireless.co.kr> on 2014/11/20 03:01:04 UTC
How to view log on yarn-client mode?
Hi,
How can I view log on yarn-client mode?
When I insert the following line on mapToPair function for example,
System.out.println("TEST TEST");
On local mode, it is displayed on console.
But on yarn-client mode, it is not on anywhere.
When I use yarn resource manager web UI, the size of 'stdout' file is 0.
And the size of 'stderr' file is non-zero, but it has only the following
lines. Maybe it's from executor launcher, but not from executor process
itself.
(I'm using Spark 1.0.0)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/grid/3/hadoop/yarn/local/filecache/10/spark-assembly-1.0.0-hadoop
2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/Static
LoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
14/11/20 10:42:29 INFO YarnSparkHadoopUtil: Using Spark's default log4j
profile: org/apache/spark/log4j-defaults.properties
14/11/20 10:42:29 INFO SecurityManager: Changing view acls to: yarn,xcapvuze
14/11/20 10:42:29 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(yarn, xcapvuze)
14/11/20 10:42:29 INFO Slf4jLogger: Slf4jLogger started
14/11/20 10:42:29 INFO Remoting: Starting remoting
14/11/20 10:42:29 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkYarnAM@cluster04:37065]
14/11/20 10:42:29 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkYarnAM@cluster04:37065]
14/11/20 10:42:29 INFO RMProxy: Connecting to ResourceManager at
cluster01/10.254.0.11:8030
14/11/20 10:42:29 INFO ExecutorLauncher: ApplicationAttemptId:
appattempt_1416441180745_0003_000001
14/11/20 10:42:29 INFO ExecutorLauncher: Registering the ApplicationMaster
14/11/20 10:42:29 INFO ExecutorLauncher: Waiting for Spark driver to be
reachable.
14/11/20 10:42:29 INFO ExecutorLauncher: Driver now available:
INNO-C-358:50050
14/11/20 10:42:29 INFO ExecutorLauncher: Listen to driver:
akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler
14/11/20 10:42:29 INFO ExecutorLauncher: Allocating 3 executors.
14/11/20 10:42:29 INFO YarnAllocationHandler: Will Allocate 3 executor
containers, each with 4480 memory
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any,
priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any,
priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any,
priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
cluster03:45454
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
cluster04:45454
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
cluster02:45454
14/11/20 10:42:30 INFO RackResolver: Resolved cluster03 to /default-rack
14/11/20 10:42:30 INFO RackResolver: Resolved cluster02 to /default-rack
14/11/20 10:42:30 INFO RackResolver: Resolved cluster04 to /default-rack
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
container_1416441180745_0003_01_000002 for on host cluster03
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
executorHostname: cluster03
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
container_1416441180745_0003_01_000004 for on host cluster02
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
executorHostname: cluster02
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
container_1416441180745_0003_01_000003 for on host cluster04
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
executorHostname: cluster04
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands:
List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p',
-Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
-Dlog4j.configuration=log4j-spark-container.properties,
org.apache.spark.executor.CoarseGrainedExecutorBackend,
akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 2, cluster02,
4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands:
List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p',
-Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
-Dlog4j.configuration=log4j-spark-container.properties,
org.apache.spark.executor.CoarseGrainedExecutorBackend,
akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 1, cluster03,
4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands:
List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p',
-Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
-Dlog4j.configuration=log4j-spark-container.properties,
org.apache.spark.executor.CoarseGrainedExecutorBackend,
akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 3, cluster04,
4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
cluster02:45454
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
cluster03:45454
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
cluster04:45454
14/11/20 10:42:30 INFO ExecutorLauncher: All executors have launched.
14/11/20 10:42:30 INFO ExecutorLauncher: Started progress reporter thread -
sleep time : 5000
14/11/20 10:43:07 INFO ExecutorLauncher: Driver terminated or disconnected!
Shutting down. Disassociated [akka.tcp://sparkYarnAM@cluster04:37065] ->
[akka.tcp://spark@INNO-C-358:50050]
14/11/20 10:43:10 INFO ExecutorLauncher: finish ApplicationMaster with
SUCCEEDED
14/11/20 10:43:10 INFO AMRMClientImpl: Waiting for application to be
successfully unregistered.
14/11/20 10:43:10 INFO ExecutorLauncher: Exited
How can I view the log?
Thanks.
Re: How to view log on yarn-client mode?
Posted by Sandy Ryza <sa...@cloudera.com>.
I agree that using "yarn logs" is cumbersome. We're working to improve
this in future releases.
On Thu, Nov 20, 2014 at 4:31 PM, innowireless TaeYun Kim <
taeyun.kim@innowireless.co.kr> wrote:
> Thank you.
>
>
>
> And setting yarn.log-aggregation-enable in yarm-site.xml to true was the
> key.
>
> It’s somewhat inconvenient that I must use ‘yarn logs’ rather than using
> YARN resource manager web UI after the app has completed (that is, it seems
> that the history server is not usable for Spark job), but it’s Ok.
>
>
>
> *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com]
> *Sent:* Thursday, November 20, 2014 2:44 PM
> *To:* innowireless TaeYun Kim
> *Cc:* user
> *Subject:* Re: How to view log on yarn-client mode?
>
>
>
> While the app is running, you can find logs from the YARN web UI by
> navigating to containers through the "Nodes" link.
>
>
>
> After the app has completed, you can use the YARN logs command:
>
> yarn logs -applicationId <your app ID>
>
>
>
> -Sandy
>
>
>
> On Wed, Nov 19, 2014 at 6:01 PM, innowireless TaeYun Kim <
> taeyun.kim@innowireless.co.kr> wrote:
>
> Hi,
>
>
>
> How can I view log on yarn-client mode?
>
> When I insert the following line on mapToPair function for example,
>
>
>
> System.out.println("TEST TEST");
>
>
>
> On local mode, it is displayed on console.
>
> But on yarn-client mode, it is not on anywhere.
>
> When I use yarn resource manager web UI, the size of ‘stdout’ file is 0.
>
> And the size of ‘stderr’ file is non-zero, but it has only the following
> lines. Maybe it’s from executor launcher, but not from executor process
> itself.
>
> (I’m using Spark 1.0.0)
>
>
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/grid/3/hadoop/yarn/local/filecache/10/spark-assembly-1.0.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.util.Shell).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> 14/11/20 10:42:29 INFO YarnSparkHadoopUtil: Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
>
> 14/11/20 10:42:29 INFO SecurityManager: Changing view acls to:
> yarn,xcapvuze
>
> 14/11/20 10:42:29 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(yarn, xcapvuze)
>
> 14/11/20 10:42:29 INFO Slf4jLogger: Slf4jLogger started
>
> 14/11/20 10:42:29 INFO Remoting: Starting remoting
>
> 14/11/20 10:42:29 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkYarnAM@cluster04:37065]
>
> 14/11/20 10:42:29 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkYarnAM@cluster04:37065]
>
> 14/11/20 10:42:29 INFO RMProxy: Connecting to ResourceManager at cluster01/
> 10.254.0.11:8030
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: ApplicationAttemptId:
> appattempt_1416441180745_0003_000001
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Registering the ApplicationMaster
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Waiting for Spark driver to be
> reachable.
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Driver now available:
> INNO-C-358:50050
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Listen to driver:
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Allocating 3 executors.
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Will Allocate 3 executor
> containers, each with 4480 memory
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster03:45454
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster04:45454
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster02:45454
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster03 to /default-rack
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster02 to /default-rack
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster04 to /default-rack
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000002 for on host cluster03
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster03
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000004 for on host cluster02
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster02
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000003 for on host cluster04
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster04
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 2,
> cluster02, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 1,
> cluster03, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 3,
> cluster04, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster02:45454
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster03:45454
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster04:45454
>
> 14/11/20 10:42:30 INFO ExecutorLauncher: All executors have launched.
>
> 14/11/20 10:42:30 INFO ExecutorLauncher: Started progress reporter thread
> - sleep time : 5000
>
> 14/11/20 10:43:07 INFO ExecutorLauncher: Driver terminated or
> disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@cluster04:37065]
> -> [akka.tcp://spark@INNO-C-358:50050]
>
> 14/11/20 10:43:10 INFO ExecutorLauncher: finish ApplicationMaster with
> SUCCEEDED
>
> 14/11/20 10:43:10 INFO AMRMClientImpl: Waiting for application to be
> successfully unregistered.
>
> 14/11/20 10:43:10 INFO ExecutorLauncher: Exited
>
>
>
> How can I view the log?
>
>
>
> Thanks.
>
>
>
RE: How to view log on yarn-client mode?
Posted by innowireless TaeYun Kim <ta...@innowireless.co.kr>.
Thank you.
And setting yarn.log-aggregation-enable in yarm-site.xml to true was the key.
It’s somewhat inconvenient that I must use ‘yarn logs’ rather than using YARN resource manager web UI after the app has completed (that is, it seems that the history server is not usable for Spark job), but it’s Ok.
From: Sandy Ryza [mailto:sandy.ryza@cloudera.com]
Sent: Thursday, November 20, 2014 2:44 PM
To: innowireless TaeYun Kim
Cc: user
Subject: Re: How to view log on yarn-client mode?
While the app is running, you can find logs from the YARN web UI by navigating to containers through the "Nodes" link.
After the app has completed, you can use the YARN logs command:
yarn logs -applicationId <your app ID>
-Sandy
On Wed, Nov 19, 2014 at 6:01 PM, innowireless TaeYun Kim <ta...@innowireless.co.kr> wrote:
Hi,
How can I view log on yarn-client mode?
When I insert the following line on mapToPair function for example,
System.out.println("TEST TEST");
On local mode, it is displayed on console.
But on yarn-client mode, it is not on anywhere.
When I use yarn resource manager web UI, the size of ‘stdout’ file is 0.
And the size of ‘stderr’ file is non-zero, but it has only the following lines. Maybe it’s from executor launcher, but not from executor process itself.
(I’m using Spark 1.0.0)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/grid/3/hadoop/yarn/local/filecache/10/spark-assembly-1.0.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/11/20 10:42:29 INFO YarnSparkHadoopUtil: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/11/20 10:42:29 INFO SecurityManager: Changing view acls to: yarn,xcapvuze
14/11/20 10:42:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, xcapvuze)
14/11/20 10:42:29 INFO Slf4jLogger: Slf4jLogger started
14/11/20 10:42:29 INFO Remoting: Starting remoting
14/11/20 10:42:29 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@cluster04:37065]
14/11/20 10:42:29 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkYarnAM@cluster04:37065]
14/11/20 10:42:29 INFO RMProxy: Connecting to ResourceManager at cluster01/10.254.0.11:8030
14/11/20 10:42:29 INFO ExecutorLauncher: ApplicationAttemptId: appattempt_1416441180745_0003_000001
14/11/20 10:42:29 INFO ExecutorLauncher: Registering the ApplicationMaster
14/11/20 10:42:29 INFO ExecutorLauncher: Waiting for Spark driver to be reachable.
14/11/20 10:42:29 INFO ExecutorLauncher: Driver now available: INNO-C-358:50050
14/11/20 10:42:29 INFO ExecutorLauncher: Listen to driver: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler
14/11/20 10:42:29 INFO ExecutorLauncher: Allocating 3 executors.
14/11/20 10:42:29 INFO YarnAllocationHandler: Will Allocate 3 executor containers, each with 4480 memory
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:4480, vCores:4>
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for : cluster03:45454
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for : cluster04:45454
14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for : cluster02:45454
14/11/20 10:42:30 INFO RackResolver: Resolved cluster03 to /default-rack
14/11/20 10:42:30 INFO RackResolver: Resolved cluster02 to /default-rack
14/11/20 10:42:30 INFO RackResolver: Resolved cluster04 to /default-rack
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container container_1416441180745_0003_01_000002 for on host cluster03
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, executorHostname: cluster03
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container container_1416441180745_0003_01_000004 for on host cluster02
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, executorHostname: cluster02
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container container_1416441180745_0003_01_000003 for on host cluster04
14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, executorHostname: cluster04
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1 file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678 timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1 file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678 timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1 file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678 timestamp: 1406511901745 type: FILE visibility: PUBLIC)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp, -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 2, cluster02, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp, -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 1, cluster03, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m , -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp, -Dlog4j.configuration=log4j-spark-container.properties, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 3, cluster04, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy : cluster02:45454
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy : cluster03:45454
14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy : cluster04:45454
14/11/20 10:42:30 INFO ExecutorLauncher: All executors have launched.
14/11/20 10:42:30 INFO ExecutorLauncher: Started progress reporter thread - sleep time : 5000
14/11/20 10:43:07 INFO ExecutorLauncher: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@cluster04:37065] -> [akka.tcp://spark@INNO-C-358:50050]
14/11/20 10:43:10 INFO ExecutorLauncher: finish ApplicationMaster with SUCCEEDED
14/11/20 10:43:10 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
14/11/20 10:43:10 INFO ExecutorLauncher: Exited
How can I view the log?
Thanks.
Re: How to view log on yarn-client mode?
Posted by Sandy Ryza <sa...@cloudera.com>.
While the app is running, you can find logs from the YARN web UI by
navigating to containers through the "Nodes" link.
After the app has completed, you can use the YARN logs command:
yarn logs -applicationId <your app ID>
-Sandy
On Wed, Nov 19, 2014 at 6:01 PM, innowireless TaeYun Kim <
taeyun.kim@innowireless.co.kr> wrote:
> Hi,
>
>
>
> How can I view log on yarn-client mode?
>
> When I insert the following line on mapToPair function for example,
>
>
>
> System.out.println("TEST TEST");
>
>
>
> On local mode, it is displayed on console.
>
> But on yarn-client mode, it is not on anywhere.
>
> When I use yarn resource manager web UI, the size of ‘stdout’ file is 0.
>
> And the size of ‘stderr’ file is non-zero, but it has only the following
> lines. Maybe it’s from executor launcher, but not from executor process
> itself.
>
> (I’m using Spark 1.0.0)
>
>
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/grid/3/hadoop/yarn/local/filecache/10/spark-assembly-1.0.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.util.Shell).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> 14/11/20 10:42:29 INFO YarnSparkHadoopUtil: Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
>
> 14/11/20 10:42:29 INFO SecurityManager: Changing view acls to:
> yarn,xcapvuze
>
> 14/11/20 10:42:29 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(yarn, xcapvuze)
>
> 14/11/20 10:42:29 INFO Slf4jLogger: Slf4jLogger started
>
> 14/11/20 10:42:29 INFO Remoting: Starting remoting
>
> 14/11/20 10:42:29 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkYarnAM@cluster04:37065]
>
> 14/11/20 10:42:29 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkYarnAM@cluster04:37065]
>
> 14/11/20 10:42:29 INFO RMProxy: Connecting to ResourceManager at cluster01/
> 10.254.0.11:8030
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: ApplicationAttemptId:
> appattempt_1416441180745_0003_000001
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Registering the ApplicationMaster
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Waiting for Spark driver to be
> reachable.
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Driver now available:
> INNO-C-358:50050
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Listen to driver:
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler
>
> 14/11/20 10:42:29 INFO ExecutorLauncher: Allocating 3 executors.
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Will Allocate 3 executor
> containers, each with 4480 memory
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:29 INFO YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:4480, vCores:4>
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster03:45454
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster04:45454
>
> 14/11/20 10:42:30 INFO AMRMClientImpl: Received new token for :
> cluster02:45454
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster03 to /default-rack
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster02 to /default-rack
>
> 14/11/20 10:42:30 INFO RackResolver: Resolved cluster04 to /default-rack
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000002 for on host cluster03
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster03
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000004 for on host cluster02
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster02
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching container
> container_1416441180745_0003_01_000003 for on host cluster04
>
> 14/11/20 10:42:30 INFO YarnAllocationHandler: Launching ExecutorRunnable.
> driverUrl: akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler,
> executorHostname: cluster04
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Starting Executor Container
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up ContainerLaunchContext
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Preparing Local resources
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Prepared Local resources
> Map(__spark__.jar -> resource { scheme: "hdfs" host: "cluster01" port: -1
> file: "/apps/spark/spark-assembly-1.0.0-hadoop2.4.0.jar" } size: 124439678
> timestamp: 1406511901745 type: FILE visibility: PUBLIC)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 2,
> cluster02, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 1,
> cluster03, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ExecutorRunnable: Setting up executor with
> commands: List({{JAVA_HOME}}/bin/java, -server,
> -XX:OnOutOfMemoryError='kill %p', -Xms4096m -Xmx4096m ,
> -Drhino.opt.level=9, -Djava.io.tmpdir={{PWD}}/tmp,
> -Dlog4j.configuration=log4j-spark-container.properties,
> org.apache.spark.executor.CoarseGrainedExecutorBackend,
> akka.tcp://spark@INNO-C-358:50050/user/CoarseGrainedScheduler, 3,
> cluster04, 4, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster02:45454
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster03:45454
>
> 14/11/20 10:42:30 INFO ContainerManagementProtocolProxy: Opening proxy :
> cluster04:45454
>
> 14/11/20 10:42:30 INFO ExecutorLauncher: All executors have launched.
>
> 14/11/20 10:42:30 INFO ExecutorLauncher: Started progress reporter thread
> - sleep time : 5000
>
> 14/11/20 10:43:07 INFO ExecutorLauncher: Driver terminated or
> disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@cluster04:37065]
> -> [akka.tcp://spark@INNO-C-358:50050]
>
> 14/11/20 10:43:10 INFO ExecutorLauncher: finish ApplicationMaster with
> SUCCEEDED
>
> 14/11/20 10:43:10 INFO AMRMClientImpl: Waiting for application to be
> successfully unregistered.
>
> 14/11/20 10:43:10 INFO ExecutorLauncher: Exited
>
>
>
> How can I view the log?
>
>
>
> Thanks.
>