You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Fengyun RAO <ra...@gmail.com> on 2015/05/14 05:37:30 UTC

--jars works in "yarn-client" but not "yarn-cluster" mode, why?

Hadoop version: CDH 5.4.

We need to connect to HBase, thus need extra
"/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
dependency.

It works in yarn-client mode:
"spark-submit --class xxx.xxx.MyApp --master yarn-client --num-executors 10
--executor-memory 10g --jars
/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
my-app.jar /input /output"

However, if we change "yarn-client" to "yarn-cluster', it throws an
ClassNotFoundException (actually the class exists in
htrace-core-3.1.0-incubating.jar):

Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
	... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)


Why --jars doesn't work in yarn-cluster mode? How to add extra
dependency in "yarn-cluster" mode?

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Fengyun RAO <ra...@gmail.com>.
Thanks, Wilfred.

The problem is, the jar "/opt/cloudera/parcels/CD
H/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
is on every node in the cluster, since we installed CDH 5.4.

thus no matter we run on client or cluster, the driver has access to the
jar.

What's more, the driver does not depend on the jar, it is the executor that
throws the "ClassNotFoundException"


2015-05-18 6:53 GMT+08:00 Wilfred Spiegelenburg <wspiegelenburg@cloudera.com
>:

> When you run the driver in the cluster the application really runs from
> the cluster and the client goes away. If the driver does not have access to
> the jars, i.e. if they are not on the cluster available somewhere, this
> will happen.
> If you run the driver on the client the driver has access to the jars
> there. Unless you have copied the jars onto the cluster it will not work.
> That is what SPARK-5377 is all about.
>
> Wilfred
>
> On 15/05/2015 00:37, Fengyun RAO wrote:
>
>> thanks, Wilfred.
>>
>> In our program, the "htrace-core-3.1.0-incubating.jar" dependency is
>> only required in the executor, not in the driver.
>> while in both "yarn-client" and "yarn-cluster", the executor runs in
>> cluster.
>>
>> and it's clearly in "yarn-cluster" mode, the jar IS in
>> "spark.yarn.secondary.jars", but still throws ClassNotFoundException
>>
>> 2015-05-14 18:52 GMT+08:00 Wilfred Spiegelenburg
>> <wspiegelenburg@cloudera.com <ma...@cloudera.com>>:
>>
>>     In the cluster the driver runs in the cluster and not locally in the
>>     spark-submit JVM. This changes what is available on your classpath.
>>     It looks like you are running into a similar situation as described
>>     in SPARK-5377.
>>
>>     Wilfred
>>
>>     On 14/05/2015 13:47, Fengyun RAO wrote:
>>
>>         I look into the "Environment" in both modes.
>>
>>         yarn-client:
>>         spark.jars
>>
>> local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar,file:/home/xxx/my-app.jar
>>
>>         yarn-cluster:
>>         spark.yarn.secondary.jars
>>
>> local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
>>
>>         I wonder why htrace exists in "spark.yarn.secondary.jars" but
>>         still not
>>         found in URLClassLoader.
>>
>>         I tried both "local" and "file" mode for the jar, still the same
>>         error.
>>
>>
>>         2015-05-14 11:37 GMT+08:00 Fengyun RAO <raofengyun@gmail.com
>>         <ma...@gmail.com>
>>         <mailto:raofengyun@gmail.com <ma...@gmail.com>>>:
>>
>>
>>
>>              Hadoop version: CDH 5.4.
>>
>>              We need to connect to HBase, thus need extra
>>
>>
>> "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
>>              dependency.
>>
>>              It works in yarn-client mode:
>>              "spark-submit --class xxx.xxx.MyApp --master yarn-client
>>              --num-executors 10 --executor-memory 10g --jars
>>
>>
>> /opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
>>              my-app.jar /input /output"
>>
>>              However, if we change "yarn-client" to "yarn-cluster', it
>>         throws an
>>              ClassNotFoundException (actually the class exists in
>>              htrace-core-3.1.0-incubating.jar):
>>
>>              Caused by: java.lang.NoClassDefFoundError:
>>         org/apache/htrace/Trace
>>                  at
>>
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>>                  at
>>
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>>                  at
>>
>> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>>                  at
>>
>> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>>                  at
>>
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>>                  at
>>
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>>                  ... 21 more
>>              Caused by: java.lang.ClassNotFoundException:
>>         org.apache.htrace.Trace
>>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>                  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>                  at java.security.AccessController.doPrivileged(Native
>>         Method)
>>                  at
>>         java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>                  at
>>         sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>                  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>>
>>              Why --jars doesn't work in yarn-cluster mode? How to add
>>         extra dependency in "yarn-cluster" mode?
>>
>>
>>
>>         --
>>
>>         ---
>>         You received this message because you are subscribed to the Google
>>         Groups "CDH Users" group.
>>         To unsubscribe from this group and stop receiving emails from
>>         it, send
>>         an email to cdh-user+unsubscribe@cloudera.org
>>         <ma...@cloudera.org>
>>         <mailto:cdh-user+unsubscribe@cloudera.org
>>         <ma...@cloudera.org>>.
>>         For more options, visit
>>         https://groups.google.com/a/cloudera.org/d/optout.
>>
>>
>>     --
>>     Wilfred Spiegelenburg
>>     Backline Customer Operations Engineer
>>     YARN/MapReduce/Spark
>>
>>     http://www.cloudera.com
>>     --
>>     http://five.sentenc.es
>>
>>     --
>>
>>     --- You received this message because you are subscribed to the
>>     Google Groups "CDH Users" group.
>>     To unsubscribe from this group and stop receiving emails from it,
>>     send an email to cdh-user+unsubscribe@cloudera.org
>>     <ma...@cloudera.org>.
>>     For more options, visit
>>     https://groups.google.com/a/cloudera.org/d/optout.
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to cdh-user+unsubscribe@cloudera.org
>> <ma...@cloudera.org>.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>
> --
> Wilfred Spiegelenburg
> Backline Customer Operations Engineer
> YARN/MapReduce/Spark
>
> http://www.cloudera.com
> --
> http://five.sentenc.es
>
> --
>
> --- You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
>

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Fengyun RAO <ra...@gmail.com>.
Thank you so much, Marcelo!

It WORKS!

2015-05-21 2:05 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:

> Hello,
>
> Sorry for the delay. The issue you're running into is because most HBase
> classes are in the system class path, while jars added with "--jars" are
> only visible to the application class loader created by Spark. So classes
> in the system class path cannot see them.
>
> You can work around this by setting "--driver-classpath
> /opt/.../htrace-core-3.1.0-incubating.jar" and "--conf
> spark.executor.extraClassPath=
> /opt/.../htrace-core-3.1.0-incubating.jar" in your spark-submit command
> line. (You can also add those configs to your spark-defaults.conf to avoid
> having to type them all the time; and don't forget to include any other
> jars that might be needed.)
>
>
> On Mon, May 18, 2015 at 11:14 PM, Fengyun RAO <ra...@gmail.com>
> wrote:
>
>> Thanks, Marcelo!
>>
>>
>> Below is the full log,
>>
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> 15/05/19 14:08:58 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
>> 15/05/19 14:08:59 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1432015548391_0003_000001
>> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to: nobody,raofengyun
>> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to: nobody,raofengyun
>> 15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody, raofengyun); users with modify permissions: Set(nobody, raofengyun)
>> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
>> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark context initialization
>> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
>> 15/05/19 14:09:00 INFO spark.SparkContext: Running Spark version 1.3.0
>> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to: nobody,raofengyun
>> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to: nobody,raofengyun
>> 15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody, raofengyun); users with modify permissions: Set(nobody, raofengyun)
>> 15/05/19 14:09:01 INFO slf4j.Slf4jLogger: Slf4jLogger started
>> 15/05/19 14:09:01 INFO Remoting: Starting remoting
>> 15/05/19 14:09:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@gs-server-v-127:7191]
>> 15/05/19 14:09:01 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@gs-server-v-127:7191]
>> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'sparkDriver' on port 7191.
>> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering MapOutputTracker
>> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering BlockManagerMaster
>> 15/05/19 14:09:01 INFO storage.DiskBlockManager: Created local directory at /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/blockmgr-3250910b-693e-46ff-b057-26d552fd8abd
>> 15/05/19 14:09:01 INFO storage.MemoryStore: MemoryStore started with capacity 259.7 MB
>> 15/05/19 14:09:01 INFO spark.HttpFileServer: HTTP File server directory is /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/httpd-5bc614bc-d8b1-473d-a807-4d9252eb679d
>> 15/05/19 14:09:01 INFO spark.HttpServer: Starting HTTP Server
>> 15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/05/19 14:09:01 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:9349
>> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'HTTP file server' on port 9349.
>> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering OutputCommitCoordinator
>> 15/05/19 14:09:01 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>> 15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
>> 15/05/19 14:09:01 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:63023
>> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'SparkUI' on port 63023.
>> 15/05/19 14:09:01 INFO ui.SparkUI: Started SparkUI at http://gs-server-v-127:63023
>> 15/05/19 14:09:02 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
>> 15/05/19 14:09:02 INFO netty.NettyBlockTransferService: Server created on 33526
>> 15/05/19 14:09:02 INFO storage.BlockManagerMaster: Trying to register BlockManager
>> 15/05/19 14:09:02 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-127:33526 with 259.7 MB RAM, BlockManagerId(<driver>, gs-server-v-127, 33526)
>> 15/05/19 14:09:02 INFO storage.BlockManagerMaster: Registered BlockManager
>> 15/05/19 14:09:02 INFO scheduler.EventLoggingListener: Logging events to hdfs://gs-server-v-127:8020/user/spark/applicationHistory/application_1432015548391_0003
>> 15/05/19 14:09:02 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@gs-server-v-127:7191/user/YarnScheduler
>> 15/05/19 14:09:02 INFO cluster.YarnClusterSchedulerBackend: ApplicationMaster registered as Actor[akka://sparkDriver/user/YarnAM#1902752386]
>> 15/05/19 14:09:02 INFO client.RMProxy: Connecting to ResourceManager at gs-server-v-127/10.200.200.56:8030
>> 15/05/19 14:09:02 INFO yarn.YarnRMClient: Registering the ApplicationMaster
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 4480 MB memory including 384 MB overhead
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:4480, vCores:1>)
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:4480, vCores:1>)
>> 15/05/19 14:09:03 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
>> 15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for : gs-server-v-127:8041
>> 15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for : gs-server-v-129:8041
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container container_1432015548391_0003_01_000002 for on host gs-server-v-127
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,  executorHostname: gs-server-v-127
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container container_1432015548391_0003_01_000003 for on host gs-server-v-129
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,  executorHostname: gs-server-v-129
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
>> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
>> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
>> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar" } size: 10759465 timestamp: 1432015733920 type: FILE visibility: PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1432015734434 type: FILE visibility: PRIVATE)
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar" } size: 10759465 timestamp: 1432015733920 type: FILE visibility: PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1432015734434 type: FILE visibility: PRIVATE)
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_LOG_URL_STDERR -> http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stderr?start=0, SPARK_DIST_CLASSPATH -> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432015548391_0003, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER -> raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1432015733920,1432015734434, SPARK_LOG_URL_STDOUT -> http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_LOG_URL_STDERR -> http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stderr?start=0, SPARK_DIST_CLASSPATH -> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432015548391_0003, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER -> raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1432015733920,1432015734434, SPARK_LOG_URL_STDOUT -> http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH", {{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191', '-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler, --executor-id, 2, --hostname, gs-server-v-129, --cores, 1, --app-id, application_1432015548391_0003, --user-class-path, file:$PWD/__app__.jar, --user-class-path, file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH", {{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191', '-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler, --executor-id, 1, --hostname, gs-server-v-127, --cores, 1, --app-id, application_1432015548391_0003, --user-class-path, file:$PWD/__app__.jar, --user-class-path, file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening proxy : gs-server-v-127:8041
>> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening proxy : gs-server-v-129:8041
>> 15/05/19 14:09:07 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@gs-server-v-127:22773/user/Executor#-351658265] with ID 1
>> 15/05/19 14:09:07 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-127:40594 with 2.1 GB RAM, BlockManagerId(1, gs-server-v-127, 40594)
>> 15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@gs-server-v-129:44560/user/Executor#-89679559] with ID 2
>> 15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
>> 15/05/19 14:09:09 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
>> 15/05/19 14:09:09 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-129:2745 with 2.1 GB RAM, BlockManagerId(2, gs-server-v-129, 2745)
>> 15/05/19 14:09:09 INFO storage.MemoryStore: ensureFreeSpace(285833) called with curMem=0, maxMem=272357130
>> 15/05/19 14:09:09 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 279.1 KB, free 259.5 MB)
>> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(22334) called with curMem=285833, maxMem=272357130
>> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.8 KB, free 259.4 MB)
>> 15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on gs-server-v-127:33526 (size: 21.8 KB, free: 259.7 MB)
>> 15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
>> 15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 0 from newAPIHadoopRDD at WdEtl.scala:56
>> 15/05/19 14:09:10 INFO spark.SparkContext: Starting job: foreach at WdEtl.scala:74
>> 15/05/19 14:09:10 INFO input.FileInputFormat: Total input paths to process : 1
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Registering RDD 1 (flatMap at WdEtl.scala:62)
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Got job 0 (foreach at WdEtl.scala:74) with 4 output partitions (allowLocal=false)
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Final stage: Stage 1(foreach at WdEtl.scala:74)
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[1] at flatMap at WdEtl.scala:62), which has no missing parents
>> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(3928) called with curMem=308167, maxMem=272357130
>> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.8 KB, free 259.4 MB)
>> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(2212) called with curMem=312095, maxMem=272357130
>> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 259.4 MB)
>> 15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on gs-server-v-127:33526 (size: 2.2 KB, free: 259.7 MB)
>> 15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
>> 15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
>> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (MapPartitionsRDD[1] at flatMap at WdEtl.scala:62)
>> 15/05/19 14:09:10 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 1 tasks
>> 15/05/19 14:09:10 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, gs-server-v-127, NODE_LOCAL, 1356 bytes)
>> 15/05/19 14:09:11 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on gs-server-v-127:40594 (size: 2.2 KB, free: 2.1 GB)
>> 15/05/19 14:09:12 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on gs-server-v-127:40594 (size: 21.8 KB, free: 2.1 GB)
>> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 87219 ms on gs-server-v-127 (1/1)
>> 15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Stage 0 (flatMap at WdEtl.scala:62) finished in 87.274 s
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: looking for newly runnable stages
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: running: Set()
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: waiting: Set(Stage 1)
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: failed: Set()
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Missing parents for Stage 1: List()
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[3] at mapPartitionsWithIndex at WdEtl.scala:64), which is now runnable
>> 15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(4728) called with curMem=314307, maxMem=272357130
>> 15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 259.4 MB)
>> 15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(2594) called with curMem=319035, maxMem=272357130
>> 15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 259.4 MB)
>> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-127:33526 (size: 2.5 KB, free: 259.7 MB)
>> 15/05/19 14:10:38 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
>> 15/05/19 14:10:38 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
>> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 1 (MapPartitionsRDD[3] at mapPartitionsWithIndex at WdEtl.scala:64)
>> 15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Adding task set 1.0 with 4 tasks
>> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-127:40594 (size: 2.5 KB, free: 2.1 GB)
>> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@gs-server-v-127:22773
>> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 148 bytes
>> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-129:2745 (size: 2.5 KB, free: 2.1 GB)
>> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@gs-server-v-129:44560
>> 15/05/19 14:10:40 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:40 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 1.0 (TID 2, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.reflect.InvocationTargetException
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>> 	... 16 more
>> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
>> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>> 	... 21 more
>> Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> 	at java.security.AccessController.doPrivileged(Native Method)
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> 	... 27 more
>>
>> 15/05/19 14:10:41 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 1.0 (TID 4, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:41 INFO scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 1]
>> 15/05/19 14:10:42 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 1.0 (TID 5, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:42 INFO scheduler.TaskSetManager: Lost task 2.0 in stage 1.0 (TID 3) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 2]
>> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 2.1 in stage 1.0 (TID 6, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 1.0 (TID 5) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 3]
>> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 1.0 (TID 7, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 1.1 in stage 1.0 (TID 4) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 4]
>> 15/05/19 14:10:44 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 1.0 (TID 8, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:44 INFO scheduler.TaskSetManager: Lost task 2.1 in stage 1.0 (TID 6) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 5]
>> 15/05/19 14:10:45 INFO scheduler.TaskSetManager: Starting task 2.2 in stage 1.0 (TID 9, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:45 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 1.0 (TID 7) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 6]
>> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 1.0 (TID 10, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 1.2 in stage 1.0 (TID 8) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 7]
>> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 1.0 (TID 11, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 2.2 in stage 1.0 (TID 9) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 8]
>> 15/05/19 14:10:47 INFO scheduler.TaskSetManager: Starting task 2.3 in stage 1.0 (TID 12, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
>> 15/05/19 14:10:47 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 1.0 (TID 10) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 9]
>> 15/05/19 14:10:47 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job
>> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Cancelling stage 1
>> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Stage 1 was cancelled
>> 15/05/19 14:10:47 INFO scheduler.DAGScheduler: Job 0 failed: foreach at WdEtl.scala:74, took 96.765394 s
>> 15/05/19 14:10:47 ERROR yarn.ApplicationMaster: User class threw exception: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.reflect.InvocationTargetException
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>> 	... 16 more
>> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
>> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>> 	... 21 more
>>
>> Driver stacktrace:
>> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.reflect.InvocationTargetException
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>> 	... 16 more
>> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
>> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>> 	... 21 more
>>
>> Driver stacktrace:
>> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> 	at scala.Option.foreach(Option.scala:236)
>> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>> 15/05/19 14:10:47 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
>> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
>> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
>> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.reflect.InvocationTargetException
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
>> 	... 16 more
>> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
>> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>> 	... 21 more
>>
>> Driver stacktrace:)
>> 15/05/19 14:10:47 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown hook
>> 15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 2.3 in stage 1.0 (TID 12, gs-server-v-127): TaskKilled (killed intentionally)
>> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
>> 15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 1.0 (TID 11, gs-server-v-129): TaskKilled (killed intentionally)
>> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
>> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
>> 15/05/19 14:10:47 INFO ui.SparkUI: Stopped Spark web UI at http://gs-server-v-127:63023
>> 15/05/19 14:10:47 INFO scheduler.DAGScheduler: Stopping DAGScheduler
>> 15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
>> 15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
>> 15/05/19 14:10:47 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
>> 15/05/19 14:10:47 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
>> 15/05/19 14:10:47 INFO storage.MemoryStore: MemoryStore cleared
>> 15/05/19 14:10:47 INFO storage.BlockManager: BlockManager stopped
>> 15/05/19 14:10:47 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
>> 15/05/19 14:10:47 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
>> 15/05/19 14:10:47 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
>> 15/05/19 14:10:47 INFO spark.SparkContext: Successfully stopped SparkContext
>>
>>
>> 2015-05-19 1:12 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:
>>
>>> On Sun, May 17, 2015 at 3:53 PM, Wilfred Spiegelenburg <
>>> wspiegelenburg@cloudera.com> wrote:
>>>
>>>> When you run the driver in the cluster the application really runs from
>>>> the cluster and the client goes away. If the driver does not have access to
>>>> the jars, i.e. if they are not on the cluster available somewhere, this
>>>> will happen.
>>>>
>>>
>>> That's not true. Files specified in "--jars" and "--files" are uploaded
>>> to the cluster before the app starts (unless they have the "local:"
>>> prefix). The visible effect on the configuration is that these files will
>>> show up in "spark.yarn.secondary.jars" as Fengyun mentioned in one of
>>> his messages.
>>>
>>> Fengyun, woule you mind sharing more than just a partial stack trace?
>>> e.g., the full driver logs would help in figuring out what's going on with
>>> that file.
>>>
>>> --
>>> Marcelo
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to cdh-user+unsubscribe@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>
>>
>>
>
>
> --
> Marcelo
>

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Marcelo Vanzin <va...@cloudera.com>.
Hello,

Sorry for the delay. The issue you're running into is because most HBase
classes are in the system class path, while jars added with "--jars" are
only visible to the application class loader created by Spark. So classes
in the system class path cannot see them.

You can work around this by setting "--driver-classpath
/opt/.../htrace-core-3.1.0-incubating.jar" and "--conf
spark.executor.extraClassPath=
/opt/.../htrace-core-3.1.0-incubating.jar" in your spark-submit command
line. (You can also add those configs to your spark-defaults.conf to avoid
having to type them all the time; and don't forget to include any other
jars that might be needed.)


On Mon, May 18, 2015 at 11:14 PM, Fengyun RAO <ra...@gmail.com> wrote:

> Thanks, Marcelo!
>
>
> Below is the full log,
>
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 15/05/19 14:08:58 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
> 15/05/19 14:08:59 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1432015548391_0003_000001
> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to: nobody,raofengyun
> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to: nobody,raofengyun
> 15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody, raofengyun); users with modify permissions: Set(nobody, raofengyun)
> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark context initialization
> 15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
> 15/05/19 14:09:00 INFO spark.SparkContext: Running Spark version 1.3.0
> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to: nobody,raofengyun
> 15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to: nobody,raofengyun
> 15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nobody, raofengyun); users with modify permissions: Set(nobody, raofengyun)
> 15/05/19 14:09:01 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/05/19 14:09:01 INFO Remoting: Starting remoting
> 15/05/19 14:09:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@gs-server-v-127:7191]
> 15/05/19 14:09:01 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@gs-server-v-127:7191]
> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'sparkDriver' on port 7191.
> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/05/19 14:09:01 INFO storage.DiskBlockManager: Created local directory at /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/blockmgr-3250910b-693e-46ff-b057-26d552fd8abd
> 15/05/19 14:09:01 INFO storage.MemoryStore: MemoryStore started with capacity 259.7 MB
> 15/05/19 14:09:01 INFO spark.HttpFileServer: HTTP File server directory is /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/httpd-5bc614bc-d8b1-473d-a807-4d9252eb679d
> 15/05/19 14:09:01 INFO spark.HttpServer: Starting HTTP Server
> 15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/05/19 14:09:01 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:9349
> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'HTTP file server' on port 9349.
> 15/05/19 14:09:01 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/05/19 14:09:01 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/05/19 14:09:01 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:63023
> 15/05/19 14:09:01 INFO util.Utils: Successfully started service 'SparkUI' on port 63023.
> 15/05/19 14:09:01 INFO ui.SparkUI: Started SparkUI at http://gs-server-v-127:63023
> 15/05/19 14:09:02 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
> 15/05/19 14:09:02 INFO netty.NettyBlockTransferService: Server created on 33526
> 15/05/19 14:09:02 INFO storage.BlockManagerMaster: Trying to register BlockManager
> 15/05/19 14:09:02 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-127:33526 with 259.7 MB RAM, BlockManagerId(<driver>, gs-server-v-127, 33526)
> 15/05/19 14:09:02 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/05/19 14:09:02 INFO scheduler.EventLoggingListener: Logging events to hdfs://gs-server-v-127:8020/user/spark/applicationHistory/application_1432015548391_0003
> 15/05/19 14:09:02 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@gs-server-v-127:7191/user/YarnScheduler
> 15/05/19 14:09:02 INFO cluster.YarnClusterSchedulerBackend: ApplicationMaster registered as Actor[akka://sparkDriver/user/YarnAM#1902752386]
> 15/05/19 14:09:02 INFO client.RMProxy: Connecting to ResourceManager at gs-server-v-127/10.200.200.56:8030
> 15/05/19 14:09:02 INFO yarn.YarnRMClient: Registering the ApplicationMaster
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 4480 MB memory including 384 MB overhead
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:4480, vCores:1>)
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:4480, vCores:1>)
> 15/05/19 14:09:03 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
> 15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for : gs-server-v-127:8041
> 15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for : gs-server-v-129:8041
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container container_1432015548391_0003_01_000002 for on host gs-server-v-127
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,  executorHostname: gs-server-v-127
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container container_1432015548391_0003_01_000003 for on host gs-server-v-129
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,  executorHostname: gs-server-v-129
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
> 15/05/19 14:09:03 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar" } size: 10759465 timestamp: 1432015733920 type: FILE visibility: PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1432015734434 type: FILE visibility: PRIVATE)
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar" } size: 10759465 timestamp: 1432015733920 type: FILE visibility: PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "gs-server-v-127" port: 8020 file: "/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1432015734434 type: FILE visibility: PRIVATE)
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_LOG_URL_STDERR -> http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stderr?start=0, SPARK_DIST_CLASSPATH -> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432015548391_0003, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER -> raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1432015733920,1432015734434, SPARK_LOG_URL_STDOUT -> http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_LOG_URL_STDERR -> http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stderr?start=0, SPARK_DIST_CLASSPATH -> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432015548391_0003, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER -> raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1432015733920,1432015734434, SPARK_LOG_URL_STDOUT -> http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH", {{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191', '-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler, --executor-id, 2, --hostname, gs-server-v-129, --cores, 1, --app-id, application_1432015548391_0003, --user-class-path, file:$PWD/__app__.jar, --user-class-path, file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH", {{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191', '-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler, --executor-id, 1, --hostname, gs-server-v-127, --cores, 1, --app-id, application_1432015548391_0003, --user-class-path, file:$PWD/__app__.jar, --user-class-path, file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening proxy : gs-server-v-127:8041
> 15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening proxy : gs-server-v-129:8041
> 15/05/19 14:09:07 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@gs-server-v-127:22773/user/Executor#-351658265] with ID 1
> 15/05/19 14:09:07 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-127:40594 with 2.1 GB RAM, BlockManagerId(1, gs-server-v-127, 40594)
> 15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@gs-server-v-129:44560/user/Executor#-89679559] with ID 2
> 15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
> 15/05/19 14:09:09 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
> 15/05/19 14:09:09 INFO storage.BlockManagerMasterActor: Registering block manager gs-server-v-129:2745 with 2.1 GB RAM, BlockManagerId(2, gs-server-v-129, 2745)
> 15/05/19 14:09:09 INFO storage.MemoryStore: ensureFreeSpace(285833) called with curMem=0, maxMem=272357130
> 15/05/19 14:09:09 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 279.1 KB, free 259.5 MB)
> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(22334) called with curMem=285833, maxMem=272357130
> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.8 KB, free 259.4 MB)
> 15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on gs-server-v-127:33526 (size: 21.8 KB, free: 259.7 MB)
> 15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
> 15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 0 from newAPIHadoopRDD at WdEtl.scala:56
> 15/05/19 14:09:10 INFO spark.SparkContext: Starting job: foreach at WdEtl.scala:74
> 15/05/19 14:09:10 INFO input.FileInputFormat: Total input paths to process : 1
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Registering RDD 1 (flatMap at WdEtl.scala:62)
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Got job 0 (foreach at WdEtl.scala:74) with 4 output partitions (allowLocal=false)
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Final stage: Stage 1(foreach at WdEtl.scala:74)
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[1] at flatMap at WdEtl.scala:62), which has no missing parents
> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(3928) called with curMem=308167, maxMem=272357130
> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.8 KB, free 259.4 MB)
> 15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(2212) called with curMem=312095, maxMem=272357130
> 15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 259.4 MB)
> 15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on gs-server-v-127:33526 (size: 2.2 KB, free: 259.7 MB)
> 15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
> 15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
> 15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (MapPartitionsRDD[1] at flatMap at WdEtl.scala:62)
> 15/05/19 14:09:10 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 1 tasks
> 15/05/19 14:09:10 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, gs-server-v-127, NODE_LOCAL, 1356 bytes)
> 15/05/19 14:09:11 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on gs-server-v-127:40594 (size: 2.2 KB, free: 2.1 GB)
> 15/05/19 14:09:12 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on gs-server-v-127:40594 (size: 21.8 KB, free: 2.1 GB)
> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 87219 ms on gs-server-v-127 (1/1)
> 15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Stage 0 (flatMap at WdEtl.scala:62) finished in 87.274 s
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: looking for newly runnable stages
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: running: Set()
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: waiting: Set(Stage 1)
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: failed: Set()
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Missing parents for Stage 1: List()
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting Stage 1 (MapPartitionsRDD[3] at mapPartitionsWithIndex at WdEtl.scala:64), which is now runnable
> 15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(4728) called with curMem=314307, maxMem=272357130
> 15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.6 KB, free 259.4 MB)
> 15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(2594) called with curMem=319035, maxMem=272357130
> 15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 259.4 MB)
> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-127:33526 (size: 2.5 KB, free: 259.7 MB)
> 15/05/19 14:10:38 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
> 15/05/19 14:10:38 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
> 15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 1 (MapPartitionsRDD[3] at mapPartitionsWithIndex at WdEtl.scala:64)
> 15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Adding task set 1.0 with 4 tasks
> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-127:40594 (size: 2.5 KB, free: 2.1 GB)
> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@gs-server-v-127:22773
> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 148 bytes
> 15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on gs-server-v-129:2745 (size: 2.5 KB, free: 2.1 GB)
> 15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to sparkExecutor@gs-server-v-129:44560
> 15/05/19 14:10:40 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:40 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 1.0 (TID 2, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> 	... 16 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
> 	... 21 more
> Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> 	... 27 more
>
> 15/05/19 14:10:41 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 1.0 (TID 4, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:41 INFO scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 1]
> 15/05/19 14:10:42 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 1.0 (TID 5, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:42 INFO scheduler.TaskSetManager: Lost task 2.0 in stage 1.0 (TID 3) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 2]
> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 2.1 in stage 1.0 (TID 6, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 1.0 (TID 5) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 3]
> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 1.0 (TID 7, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 1.1 in stage 1.0 (TID 4) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 4]
> 15/05/19 14:10:44 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 1.0 (TID 8, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:44 INFO scheduler.TaskSetManager: Lost task 2.1 in stage 1.0 (TID 6) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 5]
> 15/05/19 14:10:45 INFO scheduler.TaskSetManager: Starting task 2.2 in stage 1.0 (TID 9, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:45 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 1.0 (TID 7) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 6]
> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 1.0 (TID 10, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 1.2 in stage 1.0 (TID 8) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 7]
> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 1.0 (TID 11, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 2.2 in stage 1.0 (TID 9) on executor gs-server-v-129: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 8]
> 15/05/19 14:10:47 INFO scheduler.TaskSetManager: Starting task 2.3 in stage 1.0 (TID 12, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
> 15/05/19 14:10:47 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 1.0 (TID 10) on executor gs-server-v-127: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 9]
> 15/05/19 14:10:47 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job
> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Cancelling stage 1
> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Stage 1 was cancelled
> 15/05/19 14:10:47 INFO scheduler.DAGScheduler: Job 0 failed: foreach at WdEtl.scala:74, took 96.765394 s
> 15/05/19 14:10:47 ERROR yarn.ApplicationMaster: User class threw exception: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> 	... 16 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
> 	... 21 more
>
> Driver stacktrace:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> 	... 16 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
> 	... 21 more
>
> Driver stacktrace:
> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> 	at scala.Option.foreach(Option.scala:236)
> 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> 15/05/19 14:10:47 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException: java.lang.reflect.InvocationTargetException
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
> 	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
> 	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
> 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:64)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
> 	... 16 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
> 	... 21 more
>
> Driver stacktrace:)
> 15/05/19 14:10:47 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown hook
> 15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 2.3 in stage 1.0 (TID 12, gs-server-v-127): TaskKilled (killed intentionally)
> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
> 15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 1.0 (TID 11, gs-server-v-129): TaskKilled (killed intentionally)
> 15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
> 15/05/19 14:10:47 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
> 15/05/19 14:10:47 INFO ui.SparkUI: Stopped Spark web UI at http://gs-server-v-127:63023
> 15/05/19 14:10:47 INFO scheduler.DAGScheduler: Stopping DAGScheduler
> 15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
> 15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
> 15/05/19 14:10:47 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
> 15/05/19 14:10:47 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
> 15/05/19 14:10:47 INFO storage.MemoryStore: MemoryStore cleared
> 15/05/19 14:10:47 INFO storage.BlockManager: BlockManager stopped
> 15/05/19 14:10:47 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
> 15/05/19 14:10:47 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
> 15/05/19 14:10:47 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
> 15/05/19 14:10:47 INFO spark.SparkContext: Successfully stopped SparkContext
>
>
> 2015-05-19 1:12 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:
>
>> On Sun, May 17, 2015 at 3:53 PM, Wilfred Spiegelenburg <
>> wspiegelenburg@cloudera.com> wrote:
>>
>>> When you run the driver in the cluster the application really runs from
>>> the cluster and the client goes away. If the driver does not have access to
>>> the jars, i.e. if they are not on the cluster available somewhere, this
>>> will happen.
>>>
>>
>> That's not true. Files specified in "--jars" and "--files" are uploaded
>> to the cluster before the app starts (unless they have the "local:"
>> prefix). The visible effect on the configuration is that these files will
>> show up in "spark.yarn.secondary.jars" as Fengyun mentioned in one of
>> his messages.
>>
>> Fengyun, woule you mind sharing more than just a partial stack trace?
>> e.g., the full driver logs would help in figuring out what's going on with
>> that file.
>>
>> --
>> Marcelo
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to cdh-user+unsubscribe@cloudera.org.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>
>


-- 
Marcelo

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Fengyun RAO <ra...@gmail.com>.
Thanks, Marcelo!


Below is the full log,


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/05/19 14:08:58 INFO yarn.ApplicationMaster: Registered signal
handlers for [TERM, HUP, INT]
15/05/19 14:08:59 INFO yarn.ApplicationMaster: ApplicationAttemptId:
appattempt_1432015548391_0003_000001
15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to:
nobody,raofengyun
15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to:
nobody,raofengyun
15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(nobody, raofengyun); users with modify permissions:
Set(nobody, raofengyun)
15/05/19 14:09:00 INFO yarn.ApplicationMaster: Starting the user
application in a separate Thread
15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark
context initialization
15/05/19 14:09:00 INFO yarn.ApplicationMaster: Waiting for spark
context initialization ...
15/05/19 14:09:00 INFO spark.SparkContext: Running Spark version 1.3.0
15/05/19 14:09:00 INFO spark.SecurityManager: Changing view acls to:
nobody,raofengyun
15/05/19 14:09:00 INFO spark.SecurityManager: Changing modify acls to:
nobody,raofengyun
15/05/19 14:09:00 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view
permissions: Set(nobody, raofengyun); users with modify permissions:
Set(nobody, raofengyun)
15/05/19 14:09:01 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/05/19 14:09:01 INFO Remoting: Starting remoting
15/05/19 14:09:01 INFO Remoting: Remoting started; listening on
addresses :[akka.tcp://sparkDriver@gs-server-v-127:7191]
15/05/19 14:09:01 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkDriver@gs-server-v-127:7191]
15/05/19 14:09:01 INFO util.Utils: Successfully started service
'sparkDriver' on port 7191.
15/05/19 14:09:01 INFO spark.SparkEnv: Registering MapOutputTracker
15/05/19 14:09:01 INFO spark.SparkEnv: Registering BlockManagerMaster
15/05/19 14:09:01 INFO storage.DiskBlockManager: Created local
directory at /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/blockmgr-3250910b-693e-46ff-b057-26d552fd8abd
15/05/19 14:09:01 INFO storage.MemoryStore: MemoryStore started with
capacity 259.7 MB
15/05/19 14:09:01 INFO spark.HttpFileServer: HTTP File server
directory is /data1/cdh/yarn/nm/usercache/raofengyun/appcache/application_1432015548391_0003/httpd-5bc614bc-d8b1-473d-a807-4d9252eb679d
15/05/19 14:09:01 INFO spark.HttpServer: Starting HTTP Server
15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/05/19 14:09:01 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:9349
15/05/19 14:09:01 INFO util.Utils: Successfully started service 'HTTP
file server' on port 9349.
15/05/19 14:09:01 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/05/19 14:09:01 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/05/19 14:09:01 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/05/19 14:09:01 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:63023
15/05/19 14:09:01 INFO util.Utils: Successfully started service
'SparkUI' on port 63023.
15/05/19 14:09:01 INFO ui.SparkUI: Started SparkUI at
http://gs-server-v-127:63023
15/05/19 14:09:02 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
15/05/19 14:09:02 INFO netty.NettyBlockTransferService: Server created on 33526
15/05/19 14:09:02 INFO storage.BlockManagerMaster: Trying to register
BlockManager
15/05/19 14:09:02 INFO storage.BlockManagerMasterActor: Registering
block manager gs-server-v-127:33526 with 259.7 MB RAM,
BlockManagerId(<driver>, gs-server-v-127, 33526)
15/05/19 14:09:02 INFO storage.BlockManagerMaster: Registered BlockManager
15/05/19 14:09:02 INFO scheduler.EventLoggingListener: Logging events
to hdfs://gs-server-v-127:8020/user/spark/applicationHistory/application_1432015548391_0003
15/05/19 14:09:02 INFO yarn.ApplicationMaster: Listen to driver:
akka.tcp://sparkDriver@gs-server-v-127:7191/user/YarnScheduler
15/05/19 14:09:02 INFO cluster.YarnClusterSchedulerBackend:
ApplicationMaster registered as
Actor[akka://sparkDriver/user/YarnAM#1902752386]
15/05/19 14:09:02 INFO client.RMProxy: Connecting to ResourceManager
at gs-server-v-127/10.200.200.56:8030
15/05/19 14:09:02 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/05/19 14:09:03 INFO yarn.YarnAllocator: Will request 2 executor
containers, each with 1 cores and 4480 MB memory including 384 MB
overhead
15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:4480, vCores:1>)
15/05/19 14:09:03 INFO yarn.YarnAllocator: Container request (host:
Any, capability: <memory:4480, vCores:1>)
15/05/19 14:09:03 INFO yarn.ApplicationMaster: Started progress
reporter thread - sleep time : 5000
15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for :
gs-server-v-127:8041
15/05/19 14:09:03 INFO impl.AMRMClientImpl: Received new token for :
gs-server-v-129:8041
15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container
container_1432015548391_0003_01_000002 for on host gs-server-v-127
15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable.
driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,
 executorHostname: gs-server-v-127
15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching container
container_1432015548391_0003_01_000003 for on host gs-server-v-129
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
15/05/19 14:09:03 INFO yarn.YarnAllocator: Launching ExecutorRunnable.
driverUrl: akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,
 executorHostname: gs-server-v-129
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Starting Executor Container
15/05/19 14:09:03 INFO yarn.YarnAllocator: Received 2 containers from
YARN, launching executors on 2 of them.
15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-cached-nodemanagers-proxies : 0
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Preparing Local resources
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources
Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127"
port: 8020 file:
"/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar"
} size: 10759465 timestamp: 1432015733920 type: FILE visibility:
PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs"
host: "gs-server-v-127" port: 8020 file:
"/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar"
} size: 1475955 timestamp: 1432015734434 type: FILE visibility:
PRIVATE)
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Prepared Local resources
Map(__app__.jar -> resource { scheme: "hdfs" host: "gs-server-v-127"
port: 8020 file:
"/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar"
} size: 10759465 timestamp: 1432015733920 type: FILE visibility:
PRIVATE, htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs"
host: "gs-server-v-127" port: 8020 file:
"/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar"
} size: 1475955 timestamp: 1432015734434 type: FILE visibility:
PRIVATE)
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with
environment: Map(CLASSPATH ->
{{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*,
SPARK_LOG_URL_STDERR ->
http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stderr?start=0,
SPARK_DIST_CLASSPATH ->
/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*,
SPARK_YARN_STAGING_DIR ->
.sparkStaging/application_1432015548391_0003,
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER ->
raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
1432015733920,1432015734434, SPARK_LOG_URL_STDOUT ->
http://gs-server-v-127:8042/node/containerlogs/container_1432015548391_0003_01_000002/raofengyun/stdout?start=0,
SPARK_YARN_CACHE_FILES ->
hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with
environment: Map(CLASSPATH ->
{{PWD}}<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/assembly/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH<CPS>/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*,
SPARK_LOG_URL_STDERR ->
http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stderr?start=0,
SPARK_DIST_CLASSPATH ->
/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/hadoop/client/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/spark/conf/yarn-conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/libexec/../../hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce//.//*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hive/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/flume-ng/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../parquet/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/../avro/*:/opt/cloudera/parcels/GPLEXTRAS-5.2.0-1.cdh5.2.0.p0.20/lib/hadoop/lib/*,
SPARK_YARN_STAGING_DIR ->
.sparkStaging/application_1432015548391_0003,
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 10759465,1475955, SPARK_USER ->
raofengyun, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE,
SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
1432015733920,1432015734434, SPARK_LOG_URL_STDOUT ->
http://gs-server-v-129:8042/node/containerlogs/container_1432015548391_0003_01_000003/raofengyun/stdout?start=0,
SPARK_YARN_CACHE_FILES ->
hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/spark-wd-etl-1.0-jar-with-dependencies.jar#__app__.jar,hdfs://gs-server-v-127:8020/user/raofengyun/.sparkStaging/application_1432015548391_0003/htrace-core-3.1.0-incubating.jar#htrace-core-3.1.0-incubating.jar)
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with
commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH",
{{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p',
-Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp,
'-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191',
'-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>,
org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url,
akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,
--executor-id, 2, --hostname, gs-server-v-129, --cores, 1, --app-id,
application_1432015548391_0003, --user-class-path,
file:$PWD/__app__.jar, --user-class-path,
file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>,
<LOG_DIR>/stderr)
15/05/19 14:09:03 INFO yarn.ExecutorRunnable: Setting up executor with
commands: List(LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native:$LD_LIBRARY_PATH",
{{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p',
-Xms4096m, -Xmx4096m, -Djava.io.tmpdir={{PWD}}/tmp,
'-Dspark.shuffle.service.port=7337', '-Dspark.driver.port=7191',
'-Dspark.ui.port=0', -Dspark.yarn.app.container.log.dir=<LOG_DIR>,
org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url,
akka.tcp://sparkDriver@gs-server-v-127:7191/user/CoarseGrainedScheduler,
--executor-id, 1, --hostname, gs-server-v-127, --cores, 1, --app-id,
application_1432015548391_0003, --user-class-path,
file:$PWD/__app__.jar, --user-class-path,
file:$PWD/htrace-core-3.1.0-incubating.jar, 1>, <LOG_DIR>/stdout, 2>,
<LOG_DIR>/stderr)
15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : gs-server-v-127:8041
15/05/19 14:09:03 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : gs-server-v-129:8041
15/05/19 14:09:07 INFO cluster.YarnClusterSchedulerBackend: Registered
executor: Actor[akka.tcp://sparkExecutor@gs-server-v-127:22773/user/Executor#-351658265]
with ID 1
15/05/19 14:09:07 INFO storage.BlockManagerMasterActor: Registering
block manager gs-server-v-127:40594 with 2.1 GB RAM, BlockManagerId(1,
gs-server-v-127, 40594)
15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend: Registered
executor: Actor[akka.tcp://sparkExecutor@gs-server-v-129:44560/user/Executor#-89679559]
with ID 2
15/05/19 14:09:09 INFO cluster.YarnClusterSchedulerBackend:
SchedulerBackend is ready for scheduling beginning after reached
minRegisteredResourcesRatio: 0.8
15/05/19 14:09:09 INFO cluster.YarnClusterScheduler:
YarnClusterScheduler.postStartHook done
15/05/19 14:09:09 INFO storage.BlockManagerMasterActor: Registering
block manager gs-server-v-129:2745 with 2.1 GB RAM, BlockManagerId(2,
gs-server-v-129, 2745)
15/05/19 14:09:09 INFO storage.MemoryStore: ensureFreeSpace(285833)
called with curMem=0, maxMem=272357130
15/05/19 14:09:09 INFO storage.MemoryStore: Block broadcast_0 stored
as values in memory (estimated size 279.1 KB, free 259.5 MB)
15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(22334)
called with curMem=285833, maxMem=272357130
15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_0_piece0
stored as bytes in memory (estimated size 21.8 KB, free 259.4 MB)
15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added
broadcast_0_piece0 in memory on gs-server-v-127:33526 (size: 21.8 KB,
free: 259.7 MB)
15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of
block broadcast_0_piece0
15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 0 from
newAPIHadoopRDD at WdEtl.scala:56
15/05/19 14:09:10 INFO spark.SparkContext: Starting job: foreach at
WdEtl.scala:74
15/05/19 14:09:10 INFO input.FileInputFormat: Total input paths to process : 1
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Registering RDD 1
(flatMap at WdEtl.scala:62)
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Got job 0 (foreach at
WdEtl.scala:74) with 4 output partitions (allowLocal=false)
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Final stage: Stage
1(foreach at WdEtl.scala:74)
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Parents of final stage:
List(Stage 0)
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting Stage 0
(MapPartitionsRDD[1] at flatMap at WdEtl.scala:62), which has no
missing parents
15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(3928)
called with curMem=308167, maxMem=272357130
15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1 stored
as values in memory (estimated size 3.8 KB, free 259.4 MB)
15/05/19 14:09:10 INFO storage.MemoryStore: ensureFreeSpace(2212)
called with curMem=312095, maxMem=272357130
15/05/19 14:09:10 INFO storage.MemoryStore: Block broadcast_1_piece0
stored as bytes in memory (estimated size 2.2 KB, free 259.4 MB)
15/05/19 14:09:10 INFO storage.BlockManagerInfo: Added
broadcast_1_piece0 in memory on gs-server-v-127:33526 (size: 2.2 KB,
free: 259.7 MB)
15/05/19 14:09:10 INFO storage.BlockManagerMaster: Updated info of
block broadcast_1_piece0
15/05/19 14:09:10 INFO spark.SparkContext: Created broadcast 1 from
broadcast at DAGScheduler.scala:839
15/05/19 14:09:10 INFO scheduler.DAGScheduler: Submitting 1 missing
tasks from Stage 0 (MapPartitionsRDD[1] at flatMap at WdEtl.scala:62)
15/05/19 14:09:10 INFO cluster.YarnClusterScheduler: Adding task set
0.0 with 1 tasks
15/05/19 14:09:10 INFO scheduler.TaskSetManager: Starting task 0.0 in
stage 0.0 (TID 0, gs-server-v-127, NODE_LOCAL, 1356 bytes)
15/05/19 14:09:11 INFO storage.BlockManagerInfo: Added
broadcast_1_piece0 in memory on gs-server-v-127:40594 (size: 2.2 KB,
free: 2.1 GB)
15/05/19 14:09:12 INFO storage.BlockManagerInfo: Added
broadcast_0_piece0 in memory on gs-server-v-127:40594 (size: 21.8 KB,
free: 2.1 GB)
15/05/19 14:10:38 INFO scheduler.TaskSetManager: Finished task 0.0 in
stage 0.0 (TID 0) in 87219 ms on gs-server-v-127 (1/1)
15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Removed TaskSet
0.0, whose tasks have all completed, from pool
15/05/19 14:10:38 INFO scheduler.DAGScheduler: Stage 0 (flatMap at
WdEtl.scala:62) finished in 87.274 s
15/05/19 14:10:38 INFO scheduler.DAGScheduler: looking for newly runnable stages
15/05/19 14:10:38 INFO scheduler.DAGScheduler: running: Set()
15/05/19 14:10:38 INFO scheduler.DAGScheduler: waiting: Set(Stage 1)
15/05/19 14:10:38 INFO scheduler.DAGScheduler: failed: Set()
15/05/19 14:10:38 INFO scheduler.DAGScheduler: Missing parents for
Stage 1: List()
15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting Stage 1
(MapPartitionsRDD[3] at mapPartitionsWithIndex at WdEtl.scala:64),
which is now runnable
15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(4728)
called with curMem=314307, maxMem=272357130
15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2 stored
as values in memory (estimated size 4.6 KB, free 259.4 MB)
15/05/19 14:10:38 INFO storage.MemoryStore: ensureFreeSpace(2594)
called with curMem=319035, maxMem=272357130
15/05/19 14:10:38 INFO storage.MemoryStore: Block broadcast_2_piece0
stored as bytes in memory (estimated size 2.5 KB, free 259.4 MB)
15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added
broadcast_2_piece0 in memory on gs-server-v-127:33526 (size: 2.5 KB,
free: 259.7 MB)
15/05/19 14:10:38 INFO storage.BlockManagerMaster: Updated info of
block broadcast_2_piece0
15/05/19 14:10:38 INFO spark.SparkContext: Created broadcast 2 from
broadcast at DAGScheduler.scala:839
15/05/19 14:10:38 INFO scheduler.DAGScheduler: Submitting 4 missing
tasks from Stage 1 (MapPartitionsRDD[3] at mapPartitionsWithIndex at
WdEtl.scala:64)
15/05/19 14:10:38 INFO cluster.YarnClusterScheduler: Adding task set
1.0 with 4 tasks
15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 0.0 in
stage 1.0 (TID 1, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:38 INFO scheduler.TaskSetManager: Starting task 1.0 in
stage 1.0 (TID 2, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added
broadcast_2_piece0 in memory on gs-server-v-127:40594 (size: 2.5 KB,
free: 2.1 GB)
15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to
send map output locations for shuffle 0 to
sparkExecutor@gs-server-v-127:22773
15/05/19 14:10:38 INFO spark.MapOutputTrackerMaster: Size of output
statuses for shuffle 0 is 148 bytes
15/05/19 14:10:38 INFO storage.BlockManagerInfo: Added
broadcast_2_piece0 in memory on gs-server-v-129:2745 (size: 2.5 KB,
free: 2.1 GB)
15/05/19 14:10:38 INFO spark.MapOutputTrackerMasterActor: Asked to
send map output locations for shuffle 0 to
sparkExecutor@gs-server-v-129:44560
15/05/19 14:10:40 INFO scheduler.TaskSetManager: Starting task 2.0 in
stage 1.0 (TID 3, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:40 WARN scheduler.TaskSetManager: Lost task 1.0 in
stage 1.0 (TID 2, gs-server-v-127): java.io.IOException:
java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:64)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 16 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
	... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 27 more

15/05/19 14:10:41 INFO scheduler.TaskSetManager: Starting task 1.1 in
stage 1.0 (TID 4, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:41 INFO scheduler.TaskSetManager: Lost task 0.0 in
stage 1.0 (TID 1) on executor gs-server-v-129: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 1]
15/05/19 14:10:42 INFO scheduler.TaskSetManager: Starting task 0.1 in
stage 1.0 (TID 5, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:42 INFO scheduler.TaskSetManager: Lost task 2.0 in
stage 1.0 (TID 3) on executor gs-server-v-127: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 2]
15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 2.1 in
stage 1.0 (TID 6, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 0.1 in
stage 1.0 (TID 5) on executor gs-server-v-127: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 3]
15/05/19 14:10:43 INFO scheduler.TaskSetManager: Starting task 0.2 in
stage 1.0 (TID 7, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:43 INFO scheduler.TaskSetManager: Lost task 1.1 in
stage 1.0 (TID 4) on executor gs-server-v-129: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 4]
15/05/19 14:10:44 INFO scheduler.TaskSetManager: Starting task 1.2 in
stage 1.0 (TID 8, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:44 INFO scheduler.TaskSetManager: Lost task 2.1 in
stage 1.0 (TID 6) on executor gs-server-v-127: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 5]
15/05/19 14:10:45 INFO scheduler.TaskSetManager: Starting task 2.2 in
stage 1.0 (TID 9, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:45 INFO scheduler.TaskSetManager: Lost task 0.2 in
stage 1.0 (TID 7) on executor gs-server-v-129: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 6]
15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 0.3 in
stage 1.0 (TID 10, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 1.2 in
stage 1.0 (TID 8) on executor gs-server-v-127: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 7]
15/05/19 14:10:46 INFO scheduler.TaskSetManager: Starting task 1.3 in
stage 1.0 (TID 11, gs-server-v-129, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:46 INFO scheduler.TaskSetManager: Lost task 2.2 in
stage 1.0 (TID 9) on executor gs-server-v-129: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 8]
15/05/19 14:10:47 INFO scheduler.TaskSetManager: Starting task 2.3 in
stage 1.0 (TID 12, gs-server-v-127, PROCESS_LOCAL, 1056 bytes)
15/05/19 14:10:47 INFO scheduler.TaskSetManager: Lost task 0.3 in
stage 1.0 (TID 10) on executor gs-server-v-127: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 9]
15/05/19 14:10:47 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0
failed 4 times; aborting job
15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Cancelling stage 1
15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Stage 1 was cancelled
15/05/19 14:10:47 INFO scheduler.DAGScheduler: Job 0 failed: foreach
at WdEtl.scala:74, took 96.765394 s
15/05/19 14:10:47 ERROR yarn.ApplicationMaster: User class threw
exception: Job aborted due to stage failure: Task 0 in stage 1.0
failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID
10, gs-server-v-127): java.io.IOException:
java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:64)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 16 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
	... 21 more

Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3
in stage 1.0 (TID 10, gs-server-v-127): java.io.IOException:
java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:64)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 16 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
	... 21 more

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
15/05/19 14:10:47 INFO yarn.ApplicationMaster: Final app status:
FAILED, exitCode: 15, (reason: User class threw exception: Job aborted
due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent
failure: Lost task 0.3 in stage 1.0 (TID 10, gs-server-v-127):
java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
	at com.gridsum.spark.wd.SessionHandler.<init>(SessionHandler.scala:59)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:65)
	at com.gridsum.spark.wd.WdEtl$$anonfun$main$3.apply(WdEtl.scala:64)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:647)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:64)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 16 more
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
	... 21 more

Driver stacktrace:)
15/05/19 14:10:47 INFO yarn.ApplicationMaster: Invoking sc stop from
shutdown hook
15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 2.3 in
stage 1.0 (TID 12, gs-server-v-127): TaskKilled (killed intentionally)
15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet
1.0, whose tasks have all completed, from pool
15/05/19 14:10:47 WARN scheduler.TaskSetManager: Lost task 1.3 in
stage 1.0 (TID 11, gs-server-v-129): TaskKilled (killed intentionally)
15/05/19 14:10:47 INFO cluster.YarnClusterScheduler: Removed TaskSet
1.0, whose tasks have all completed, from pool
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
15/05/19 14:10:47 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
15/05/19 14:10:47 INFO ui.SparkUI: Stopped Spark web UI at
http://gs-server-v-127:63023
15/05/19 14:10:47 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Shutting
down all executors
15/05/19 14:10:47 INFO cluster.YarnClusterSchedulerBackend: Asking
each executor to shut down
15/05/19 14:10:47 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
OutputCommitCoordinator stopped!
15/05/19 14:10:47 INFO spark.MapOutputTrackerMasterActor:
MapOutputTrackerActor stopped!
15/05/19 14:10:47 INFO storage.MemoryStore: MemoryStore cleared
15/05/19 14:10:47 INFO storage.BlockManager: BlockManager stopped
15/05/19 14:10:47 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
15/05/19 14:10:47 INFO
remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote
daemon.
15/05/19 14:10:47 INFO
remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut
down; proceeding with flushing remote transports.
15/05/19 14:10:47 INFO spark.SparkContext: Successfully stopped SparkContext


2015-05-19 1:12 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:

> On Sun, May 17, 2015 at 3:53 PM, Wilfred Spiegelenburg <
> wspiegelenburg@cloudera.com> wrote:
>
>> When you run the driver in the cluster the application really runs from
>> the cluster and the client goes away. If the driver does not have access to
>> the jars, i.e. if they are not on the cluster available somewhere, this
>> will happen.
>>
>
> That's not true. Files specified in "--jars" and "--files" are uploaded to
> the cluster before the app starts (unless they have the "local:" prefix).
> The visible effect on the configuration is that these files will show up in
> "spark.yarn.secondary.jars" as Fengyun mentioned in one of his messages.
>
> Fengyun, woule you mind sharing more than just a partial stack trace?
> e.g., the full driver logs would help in figuring out what's going on with
> that file.
>
> --
> Marcelo
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
>

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Fengyun RAO <ra...@gmail.com>.
thanks, Wilfred.

In our program, the "htrace-core-3.1.0-incubating.jar" dependency is only
required in the executor, not in the driver.
while in both "yarn-client" and "yarn-cluster", the executor runs in
cluster.

and it's clearly in "yarn-cluster" mode, the jar IS in
"spark.yarn.secondary.jars",
but still throws ClassNotFoundException

2015-05-14 18:52 GMT+08:00 Wilfred Spiegelenburg <
wspiegelenburg@cloudera.com>:

> In the cluster the driver runs in the cluster and not locally in the
> spark-submit JVM. This changes what is available on your classpath. It
> looks like you are running into a similar situation as described in
> SPARK-5377.
>
> Wilfred
>
> On 14/05/2015 13:47, Fengyun RAO wrote:
>
>> I look into the "Environment" in both modes.
>>
>> yarn-client:
>> spark.jars
>>
>> local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar,file:/home/xxx/my-app.jar
>>
>> yarn-cluster:
>> spark.yarn.secondary.jars
>>
>> local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
>>
>> I wonder why htrace exists in "spark.yarn.secondary.jars" but still not
>> found in URLClassLoader.
>>
>> I tried both "local" and "file" mode for the jar, still the same error.
>>
>>
>> 2015-05-14 11:37 GMT+08:00 Fengyun RAO <raofengyun@gmail.com
>> <ma...@gmail.com>>:
>>
>>
>>     Hadoop version: CDH 5.4.
>>
>>     We need to connect to HBase, thus need extra
>>
>> "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
>>     dependency.
>>
>>     It works in yarn-client mode:
>>     "spark-submit --class xxx.xxx.MyApp --master yarn-client
>>     --num-executors 10 --executor-memory 10g --jars
>>
>> /opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
>>     my-app.jar /input /output"
>>
>>     However, if we change "yarn-client" to "yarn-cluster', it throws an
>>     ClassNotFoundException (actually the class exists in
>>     htrace-core-3.1.0-incubating.jar):
>>
>>     Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
>>         at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
>>         at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
>>         at
>> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
>>         at
>> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
>>         at
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
>>         at
>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
>>         ... 21 more
>>     Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>>
>>     Why --jars doesn't work in yarn-cluster mode? How to add extra
>> dependency in "yarn-cluster" mode?
>>
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to cdh-user+unsubscribe@cloudera.org
>> <ma...@cloudera.org>.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>
> --
> Wilfred Spiegelenburg
> Backline Customer Operations Engineer
> YARN/MapReduce/Spark
>
> http://www.cloudera.com
> --
> http://five.sentenc.es
>
> --
>
> --- You received this message because you are subscribed to the Google
> Groups "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
>

Re: --jars works in "yarn-client" but not "yarn-cluster" mode, why?

Posted by Fengyun RAO <ra...@gmail.com>.
I look into the "Environment" in both modes.

yarn-client:
spark.jars
local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar,file:/home/xxx/my-app.jar
yarn-cluster:
spark.yarn.secondary.jars
local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
I wonder why htrace exists in "spark.yarn.secondary.jars" but still not
found in URLClassLoader.

I tried both "local" and "file" mode for the jar, still the same error.


2015-05-14 11:37 GMT+08:00 Fengyun RAO <ra...@gmail.com>:

> Hadoop version: CDH 5.4.
>
> We need to connect to HBase, thus need extra
> "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar"
> dependency.
>
> It works in yarn-client mode:
> "spark-submit --class xxx.xxx.MyApp --master yarn-client --num-executors
> 10 --executor-memory 10g --jars
> /opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar
> my-app.jar /input /output"
>
> However, if we change "yarn-client" to "yarn-cluster', it throws an
> ClassNotFoundException (actually the class exists in
> htrace-core-3.1.0-incubating.jar):
>
> Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
> 	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)
> 	at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
> 	at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
> 	at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850)
> 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635)
> 	... 21 more
> Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>
> Why --jars doesn't work in yarn-cluster mode? How to add extra dependency in "yarn-cluster" mode?
>
>
>