You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by liang <pw...@gmail.com> on 2019/12/12 07:45:16 UTC
Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
Hi there,
I'm trying to build cube with Spark. I've followed the instructions in this
tutorial: Build Cube with Spark
<http://kylin.apache.org/docs/tutorial/cube_spark.html>. But the job failed
at step: "Convert Cuboid Data to HFile". The logs shows
"ClassNotFoundException:
org.apache.hadoop.hbase.io.MetricsIOWrapper"
org.apache.kylin.engine.spark.exception.SparkException: OS command error
> exit with return code: 1, error message: 2019-12-12 15:10:46 WARN
> SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future.
> Please use the new key 'spark.executor.memoryOverhead' instead.
> 2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at
> host/10.16.15.15:9032
> 2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from
> cluster with 150 NodeManagers
> 2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not
> requested more than the maximum memory capability of the cluster (49150 MB
> per container)
> 2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with
> 2432 MB memory including 384 MB overhead
> 2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context
> for our AM
> 2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment
> for our AM container
> 2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM
> container
> 2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems
> are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar
> 2019-12-12 15:10:50 INFO Client:54 - Uploading resource
> file:/usr/local/kylin/lib/kylin-job-2.6.4.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar
> 2019-12-12 15:10:52 WARN Client:66 - Same path resource
> file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added
> multiple times to distributed cache.
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar
> 2019-12-12 15:10:53 INFO Client:54 - Uploading resource
> file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip
> ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to:
> hadoop
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to:
> hadoop
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups
> to:
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls groups
> to:
> 2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hadoop); groups with view permissions: Set(); users with modify
> permissions: Set(hadoop); groups with modify permissions: Set()
> 2019-12-12 15:10:54 INFO Client:54 - Submitting application
> application_1549925837808_10474661 to ResourceManager
> 2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application
> application_1549925837808_10474661
> 2019-12-12 15:10:55 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:55 INFO Client:54 -
> client token: N/A
> diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the
> scheduler and is not yet activated. (Resource request: <memory:3072,
> vCores:1> exceeds current queue or its parents maximum resource allowed).
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:10:56 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:57 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:58 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:59 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:00 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:01 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:02 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:03 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:04 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:05 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:06 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:07 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:08 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:09 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:10 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:11 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:11 INFO Client:54 -
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 10.16.15.244
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:12 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:13 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:14 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:15 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:16 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:17 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:18 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:19 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:20 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:21 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:22 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:23 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:24 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:25 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:26 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:27 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:28 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:29 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:30 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:31 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:32 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:32 INFO Client:54 -
> client token: N/A
> diagnostics: AM container is launched, waiting for AM container to
> Register with RM
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:33 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:34 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:35 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:36 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:37 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:38 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:39 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:40 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:41 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:42 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:42 INFO Client:54 -
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 10.16.15.59
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:43 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:44 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:45 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:46 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:47 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:48 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:49 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:50 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:51 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:52 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:53 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:54 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:55 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:56 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:57 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:58 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:59 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:00 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:01 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:02 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:03 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:04 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:05 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:06 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:07 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:08 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:09 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: FINISHED)
> 2019-12-12 15:12:09 INFO Client:54 -
> client token: N/A
> diagnostics: User class threw exception: java.lang.RuntimeException: error
> execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause:
> Job aborted.
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
> Caused by: org.apache.spark.SparkException: Job aborted.
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
> at
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task
> 0.3 in stage 1.0 (TID 5, host.dns.example.com, executor 15):
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/io/MetricsIOWrapper
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
> at
> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.MetricsIOWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 16 more
>
> Driver stacktrace:
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
> ... 16 more
> Caused by: org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/io/MetricsIOWrapper
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
> at
> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.MetricsIOWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 16 more
>
> ApplicationMaster host: 10.16.15.59
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: FAILED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> Exception in thread "main" org.apache.spark.SparkException: Application
> application_1549925837808_10474661 finished with failed status
> at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
> at
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1
> The command is:
> export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop &&
> /usr/local/spark/bin/spark-submit --class
> org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40
> --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf
> spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
> spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
> --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn
> --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf
> spark.hadoop.yarn.timeline-service.enabled=false --conf
> spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
> spark.eventLog.dir=hdfs:///kylin/spark-history --conf
> spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1
> --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false
> --conf spark.yarn.executor.memoryOverhead=1024 --conf
> spark.hadoop.dfs.replication=2 --conf
> spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G
> --conf spark.submit.deployMode=cluster --conf
> spark.shuffle.service.enabled=true --jars
> /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar,
> /usr/local/kylin/lib/kylin-job-2.6.4.jar -className
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile
> -counterOutput
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter
> -cubename test1 -output
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile
> -input
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/
> -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata
> -hbaseConfPath
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
This error message is very like the tips on the tutorial which I mentioned
above. I've tried to put hbase-hadoop-compat-1.1.5.jar and
hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I
found there's no spark folder under the $KYLIN_HOME. This is strange
because the tutorial said Kylin has a built-in Spark. Maybe the docs are
out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put
these two jars into the "jars" folder. But when after I resume the job,
that error happened again. Nothing helps.
I've also tried to put these two jars under the $SPARK_HOME/jars. But it
still doesn't work.
These versions are my current setup:
Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz
Spark: 2.3.2
HBase: 1.1.5
Hadoop: 2.7.2
Thanks,
Liang
Re: Build Cube with Spark Failed at Step: Convert Cuboid Data to
HFile
Posted by nichunen <ni...@apache.org>.
Hi Liang,
Glad to hear that, and thanks for your sharing.
Best regards,
Ni Chunen / George
On 12/13/2019 18:39，liang<pw...@gmail.com> wrote：
Hi all,
I've resolved the issue. Someone deployed non-official Apache Spark on the Kylin building server. There are a lot of higher versions of HBase jars under the $SPARK_HOME/jars.
-rwxr-xr-x 1 hadoop hadoop 1407871 Oct 10 14:12 hbase-client-1.1.2.2.6.5.0-292.jar
-rwxr-xr-x 1 hadoop hadoop 577452 Oct 10 14:12 hbase-common-1.1.2.2.6.5.0-292.jar
-rwxr-xr-x 1 hadoop hadoop 4988181 Oct 10 14:12 hbase-protocol-1.1.2.2.6.5.0-292.jar
-rwxr-xr-x 1 hadoop hadoop 4644937 Oct 10 14:12 hbase-server-1.1.2.2.6.5.0-292.jar
-rwxr-xr-x 1 hadoop hadoop 110687 Oct 10 14:12 hive-hbase-handler-2.0.0.jar
After deleted these jars, the cube can be built with Spark successfully.
Thanks,
Liang
On Thu, Dec 12, 2019 at 7:36 PM liang <pw...@gmail.com> wrote:
Hi Rupeng,
Here's the hadoop version:
-----------
hadoop version
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /usr/local/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
------------
Thanks,
Liang
On Thu, Dec 12, 2019 at 7:15 PM Wang rupeng <wa...@live.cn> wrote:
Hi,
It seems like environment problem. What’s your cdh or hdp version?
-------------------
Best wishes,
Rupeng Wang
发件人: liang <pw...@gmail.com>
答复: <us...@kylin.apache.org>
日期: 2019年12月12日星期四 15:46
收件人: <us...@kylin.apache.org>
主题: Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
Hi there,
I'm trying to build cube with Spark. I've followed the instructions in this tutorial: Build Cube with Spark. But the job failed at step: "Convert Cuboid Data to HFile". The logs shows "ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper"
org.apache.kylin.engine.spark.exception.SparkException: OS command error exit with return code: 1, error message: 2019-12-12 15:10:46 WARN SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at host/10.16.15.15:9032
2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from cluster with 150 NodeManagers
2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (49150 MB per container)
2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with 2432 MB memory including 384 MB overhead
2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context for our AM
2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment for our AM container
2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM container
2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar
2019-12-12 15:10:50 INFO Client:54 - Uploading resource file:/usr/local/kylin/lib/kylin-job-2.6.4.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar
2019-12-12 15:10:52 WARN Client:66 - Same path resource file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added multiple times to distributed cache.
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar
2019-12-12 15:10:53 INFO Client:54 - Uploading resource file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to: hadoop
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to: hadoop
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups to:
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls groups to:
2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
2019-12-12 15:10:54 INFO Client:54 - Submitting application application_1549925837808_10474661 to ResourceManager
2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application application_1549925837808_10474661
2019-12-12 15:10:55 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:55 INFO Client:54 -
client token: N/A
diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the scheduler and is not yet activated. (Resource request: <memory:3072, vCores:1> exceeds current queue or its parents maximum resource allowed).
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:10:56 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:57 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:58 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:59 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:00 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:01 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:02 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:03 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:04 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:05 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:06 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:07 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:08 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:09 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:10 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:11 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:11 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.16.15.244
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:12 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:13 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:14 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:15 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:16 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:17 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:18 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:19 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:20 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:21 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:22 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:23 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:24 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:25 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:26 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:27 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:28 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:29 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:30 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:31 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:32 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:32 INFO Client:54 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:33 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:34 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:35 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:36 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:37 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:38 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:39 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:40 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:41 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:42 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:42 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.16.15.59
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:43 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:44 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:45 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:46 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:47 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:48 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:49 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:50 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:51 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:52 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:53 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:54 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:55 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:56 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:57 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:58 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:59 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:00 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:01 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:02 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:03 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:04 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:05 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:06 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:07 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:08 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:09 INFO Client:54 - Application report for application_1549925837808_10474661 (state: FINISHED)
2019-12-12 15:12:09 INFO Client:54 -
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted.
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
at org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 6 more
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 5, host.dns.example.com, executor 15): org.apache.spark.SparkException: Task failed while writing rows
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/io/MetricsIOWrapper
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)
at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
... 16 more
Caused by: org.apache.spark.SparkException: Task failed while writing rows
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/io/MetricsIOWrapper
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
ApplicationMaster host: 10.16.15.59
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: FAILED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
Exception in thread "main" org.apache.spark.SparkException: Application application_1549925837808_10474661 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1
The command is:
export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop && /usr/local/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40 --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1 --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.hadoop.dfs.replication=2 --conf spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G --conf spark.submit.deployMode=cluster --conf spark.shuffle.service.enabled=true --jars /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar, /usr/local/kylin/lib/kylin-job-2.6.4.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile -counterOutput hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter -cubename test1 -output hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile -input hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/ -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata -hbaseConfPath hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml
at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This error message is very like the tips on the tutorial which I mentioned above. I've tried to put hbase-hadoop-compat-1.1.5.jar and hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I found there's no spark folder under the $KYLIN_HOME. This is strange because the tutorial said Kylin has a built-in Spark. Maybe the docs are out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put these two jars into the "jars" folder. But when after I resume the job, that error happened again. Nothing helps.
I've also tried to put these two jars under the $SPARK_HOME/jars. But it still doesn't work.
These versions are my current setup:
Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz
Spark: 2.3.2
HBase: 1.1.5
Hadoop: 2.7.2
Thanks,
Liang
Re: Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
Posted by liang <pw...@gmail.com>.
Hi all,
I've resolved the issue. Someone deployed non-official Apache Spark on the
Kylin building server. There are a lot of higher versions of HBase jars
under the $SPARK_HOME/jars.
-rwxr-xr-x 1 hadoop hadoop 1407871 Oct 10 14:12
> hbase-client-1.1.2.2.6.5.0-292.jar
> -rwxr-xr-x 1 hadoop hadoop 577452 Oct 10 14:12
> hbase-common-1.1.2.2.6.5.0-292.jar
> -rwxr-xr-x 1 hadoop hadoop 4988181 Oct 10 14:12
> hbase-protocol-1.1.2.2.6.5.0-292.jar
> -rwxr-xr-x 1 hadoop hadoop 4644937 Oct 10 14:12
> hbase-server-1.1.2.2.6.5.0-292.jar
> -rwxr-xr-x 1 hadoop hadoop 110687 Oct 10 14:12
> hive-hbase-handler-2.0.0.jar
>
After deleted these jars, the cube can be built with Spark successfully.
Thanks,
Liang
On Thu, Dec 12, 2019 at 7:36 PM liang <pw...@gmail.com> wrote:
> Hi Rupeng,
>
> Here's the hadoop version:
> -----------
> hadoop version
> Hadoop 2.7.2
> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> b165c4fe8a74265c792ce23f546c64604acf0e41
> Compiled by jenkins on 2016-01-26T00:08Z
> Compiled with protoc 2.5.0
> From source with checksum d0fda26633fa762bff87ec759ebe689c
> This command was run using
> /usr/local/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
> ------------
>
> Thanks,
> Liang
>
>
>
> On Thu, Dec 12, 2019 at 7:15 PM Wang rupeng <wa...@live.cn> wrote:
>
>> Hi,
>>
>> It seems like environment problem. What’s your cdh or hdp version?
>>
>>
>>
>>
>>
>> -------------------
>>
>> Best wishes,
>>
>> Rupeng Wang
>>
>>
>>
>>
>>
>> *发件人**: *liang <pw...@gmail.com>
>> *答复**: *<us...@kylin.apache.org>
>> *日期**: *2019年12月12日 星期四 15:46
>> *收件人**: *<us...@kylin.apache.org>
>> *主题**: *Build Cube with Spark Failed at Step: Convert Cuboid Data to
>> HFile
>>
>>
>>
>> Hi there,
>>
>>
>>
>> I'm trying to build cube with Spark. I've followed the instructions in
>> this tutorial: Build Cube with Spark
>> <http://kylin.apache.org/docs/tutorial/cube_spark.html>. But the job
>> failed at step: "Convert Cuboid Data to HFile". The logs shows "ClassNotFoundException:
>> org.apache.hadoop.hbase.io.MetricsIOWrapper"
>>
>>
>>
>> org.apache.kylin.engine.spark.exception.SparkException: OS command error
>> exit with return code: 1, error message: 2019-12-12 15:10:46 WARN
>> SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead'
>> has been deprecated as of Spark 2.3 and may be removed in the future.
>> Please use the new key 'spark.executor.memoryOverhead' instead.
>> 2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at
>> host/10.16.15.15:9032
>> 2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from
>> cluster with 150 NodeManagers
>> 2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not
>> requested more than the maximum memory capability of the cluster (49150 MB
>> per container)
>> 2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with
>> 2432 MB memory including 384 MB overhead
>> 2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context
>> for our AM
>> 2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment
>> for our AM container
>> 2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM
>> container
>> 2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems
>> are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar
>> 2019-12-12 15:10:50 INFO Client:54 - Uploading resource
>> file:/usr/local/kylin/lib/kylin-job-2.6.4.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar
>> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar
>> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar
>> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar
>> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar
>> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar
>> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
>> file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar
>> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar
>> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar
>> 2019-12-12 15:10:52 WARN Client:66 - Same path resource
>> file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added
>> multiple times to distributed cache.
>> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
>> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar
>> 2019-12-12 15:10:53 INFO Client:54 - Uploading resource
>> file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip
>> ->
>> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip
>> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to:
>> hadoop
>> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to:
>> hadoop
>> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups
>> to:
>> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls
>> groups to:
>> 2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(hadoop); groups with view permissions: Set(); users with modify
>> permissions: Set(hadoop); groups with modify permissions: Set()
>> 2019-12-12 15:10:54 INFO Client:54 - Submitting application
>> application_1549925837808_10474661 to ResourceManager
>> 2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application
>> application_1549925837808_10474661
>> 2019-12-12 15:10:55 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:10:55 INFO Client:54 -
>> client token: N/A
>> diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the
>> scheduler and is not yet activated. (Resource request: <memory:3072,
>> vCores:1> exceeds current queue or its parents maximum resource allowed).
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: root.hadoop
>> start time: 1576134654497
>> final status: UNDEFINED
>> tracking URL:
>> http://host.example.com:8981/proxy/application_1549925837808_10474661/
>> user: hadoop
>> 2019-12-12 15:10:56 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:10:57 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:10:58 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:10:59 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:00 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:01 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:02 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:03 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:04 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:05 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:06 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:07 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:08 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:09 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:10 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:11 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:11 INFO Client:54 -
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: 10.16.15.244
>> ApplicationMaster RPC port: 0
>> queue: root.hadoop
>> start time: 1576134654497
>> final status: UNDEFINED
>> tracking URL:
>> http://host.example.com:8981/proxy/application_1549925837808_10474661/
>> user: hadoop
>> 2019-12-12 15:11:12 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:13 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:14 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:15 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:16 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:17 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:18 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:19 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:20 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:21 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:22 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:23 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:24 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:25 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:26 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:27 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:28 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:29 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:30 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:31 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:32 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:32 INFO Client:54 -
>> client token: N/A
>> diagnostics: AM container is launched, waiting for AM container to
>> Register with RM
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: root.hadoop
>> start time: 1576134654497
>> final status: UNDEFINED
>> tracking URL:
>> http://host.example.com:8981/proxy/application_1549925837808_10474661/
>> user: hadoop
>> 2019-12-12 15:11:33 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:34 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:35 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:36 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:37 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:38 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:39 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:40 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:41 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: ACCEPTED)
>> 2019-12-12 15:11:42 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:42 INFO Client:54 -
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: 10.16.15.59
>> ApplicationMaster RPC port: 0
>> queue: root.hadoop
>> start time: 1576134654497
>> final status: UNDEFINED
>> tracking URL:
>> http://host.example.com:8981/proxy/application_1549925837808_10474661/
>> user: hadoop
>> 2019-12-12 15:11:43 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:44 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:45 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:46 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:47 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:48 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:49 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:50 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:51 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:52 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:53 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:54 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:55 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:56 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:57 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:58 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:11:59 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:00 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:01 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:02 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:03 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:04 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:05 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:06 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:07 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:08 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: RUNNING)
>> 2019-12-12 15:12:09 INFO Client:54 - Application report for
>> application_1549925837808_10474661 (state: FINISHED)
>> 2019-12-12 15:12:09 INFO Client:54 -
>> client token: N/A
>> diagnostics: User class threw exception: java.lang.RuntimeException:
>> error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root
>> cause: Job aborted.
>> at
>> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
>> Caused by: org.apache.spark.SparkException: Job aborted.
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
>> at
>> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
>> at
>> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
>> at
>> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>> at
>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
>> at
>> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
>> at
>> org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
>> at
>> org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238)
>> at
>> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
>> ... 6 more
>> Caused by: org.apache.spark.SparkException: Job aborted due to stage
>> failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task
>> 0.3 in stage 1.0 (TID 5, host.dns.example.com, executor 15):
>> org.apache.spark.SparkException: Task failed while writing rows
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>> at org.apache.spark.scheduler.Task.run(Task.scala:109)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hbase/io/MetricsIOWrapper
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>> at
>> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
>> at
>> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
>> ... 8 more
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.MetricsIOWrapper
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 16 more
>>
>> Driver stacktrace:
>> at org.apache.spark.scheduler.DAGScheduler.org
>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
>> at
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>> at
>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
>> at scala.Option.foreach(Option.scala:257)
>> at
>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
>> ... 16 more
>> Caused by: org.apache.spark.SparkException: Task failed while writing rows
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>> at org.apache.spark.scheduler.Task.run(Task.scala:109)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hbase/io/MetricsIOWrapper
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>> at
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>> at
>> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
>> at
>> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
>> at
>> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
>> ... 8 more
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.MetricsIOWrapper
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 16 more
>>
>> ApplicationMaster host: 10.16.15.59
>> ApplicationMaster RPC port: 0
>> queue: root.hadoop
>> start time: 1576134654497
>> final status: FAILED
>> tracking URL:
>> http://host.example.com:8981/proxy/application_1549925837808_10474661/
>> user: hadoop
>> Exception in thread "main" org.apache.spark.SparkException: Application
>> application_1549925837808_10474661 finished with failed status
>> at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
>> at
>> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called
>> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
>> /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409
>> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
>> /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1
>> The command is:
>> export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop &&
>> /usr/local/spark/bin/spark-submit --class
>> org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40
>> --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf
>> spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
>> spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
>> --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn
>> --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf
>> spark.hadoop.yarn.timeline-service.enabled=false --conf
>> spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
>> spark.eventLog.dir=hdfs:///kylin/spark-history --conf
>> spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1
>> --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false
>> --conf spark.yarn.executor.memoryOverhead=1024 --conf
>> spark.hadoop.dfs.replication=2 --conf
>> spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G
>> --conf spark.submit.deployMode=cluster --conf
>> spark.shuffle.service.enabled=true --jars
>> /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar,
>> /usr/local/kylin/lib/kylin-job-2.6.4.jar -className
>> org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
>> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile
>> -counterOutput
>> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter
>> -cubename test1 -output
>> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile
>> -input
>> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/
>> -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata
>> -hbaseConfPath
>> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml
>> at
>> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
>> at
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
>> at
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
>> at
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
>> at
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> This error message is very like the tips on the tutorial which I
>> mentioned above. I've tried to put hbase-hadoop-compat-1.1.5.jar and
>> hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I
>> found there's no spark folder under the $KYLIN_HOME. This is strange
>> because the tutorial said Kylin has a built-in Spark. Maybe the docs are
>> out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put
>> these two jars into the "jars" folder. But when after I resume the job,
>> that error happened again. Nothing helps.
>> I've also tried to put these two jars under the $SPARK_HOME/jars. But it
>> still doesn't work.
>>
>>
>>
>> These versions are my current setup:
>>
>> Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz
>>
>> Spark: 2.3.2
>>
>> HBase: 1.1.5
>>
>> Hadoop: 2.7.2
>>
>>
>>
>> Thanks,
>>
>> Liang
>>
>>
>>
>>
>>
>
Re: Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
Posted by liang <pw...@gmail.com>.
Hi Rupeng,
Here's the hadoop version:
-----------
hadoop version
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using
/usr/local/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
------------
Thanks,
Liang
On Thu, Dec 12, 2019 at 7:15 PM Wang rupeng <wa...@live.cn> wrote:
> Hi,
>
> It seems like environment problem. What’s your cdh or hdp version?
>
>
>
>
>
> -------------------
>
> Best wishes,
>
> Rupeng Wang
>
>
>
>
>
> *发件人**: *liang <pw...@gmail.com>
> *答复**: *<us...@kylin.apache.org>
> *日期**: *2019年12月12日 星期四 15:46
> *收件人**: *<us...@kylin.apache.org>
> *主题**: *Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
>
>
>
> Hi there,
>
>
>
> I'm trying to build cube with Spark. I've followed the instructions in
> this tutorial: Build Cube with Spark
> <http://kylin.apache.org/docs/tutorial/cube_spark.html>. But the job
> failed at step: "Convert Cuboid Data to HFile". The logs shows "ClassNotFoundException:
> org.apache.hadoop.hbase.io.MetricsIOWrapper"
>
>
>
> org.apache.kylin.engine.spark.exception.SparkException: OS command error
> exit with return code: 1, error message: 2019-12-12 15:10:46 WARN
> SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future.
> Please use the new key 'spark.executor.memoryOverhead' instead.
> 2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at
> host/10.16.15.15:9032
> 2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from
> cluster with 150 NodeManagers
> 2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not
> requested more than the maximum memory capability of the cluster (49150 MB
> per container)
> 2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with
> 2432 MB memory including 384 MB overhead
> 2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context
> for our AM
> 2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment
> for our AM container
> 2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM
> container
> 2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems
> are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar
> 2019-12-12 15:10:50 INFO Client:54 - Uploading resource
> file:/usr/local/kylin/lib/kylin-job-2.6.4.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar
> 2019-12-12 15:10:51 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar
> 2019-12-12 15:10:52 WARN Client:66 - Same path resource
> file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added
> multiple times to distributed cache.
> 2019-12-12 15:10:52 INFO Client:54 - Uploading resource
> file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar
> 2019-12-12 15:10:53 INFO Client:54 - Uploading resource
> file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip
> ->
> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to:
> hadoop
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to:
> hadoop
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups
> to:
> 2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls groups
> to:
> 2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(hadoop); groups with view permissions: Set(); users with modify
> permissions: Set(hadoop); groups with modify permissions: Set()
> 2019-12-12 15:10:54 INFO Client:54 - Submitting application
> application_1549925837808_10474661 to ResourceManager
> 2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application
> application_1549925837808_10474661
> 2019-12-12 15:10:55 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:55 INFO Client:54 -
> client token: N/A
> diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the
> scheduler and is not yet activated. (Resource request: <memory:3072,
> vCores:1> exceeds current queue or its parents maximum resource allowed).
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:10:56 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:57 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:58 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:10:59 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:00 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:01 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:02 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:03 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:04 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:05 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:06 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:07 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:08 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:09 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:10 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:11 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:11 INFO Client:54 -
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 10.16.15.244
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:12 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:13 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:14 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:15 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:16 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:17 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:18 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:19 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:20 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:21 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:22 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:23 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:24 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:25 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:26 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:27 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:28 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:29 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:30 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:31 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:32 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:32 INFO Client:54 -
> client token: N/A
> diagnostics: AM container is launched, waiting for AM container to
> Register with RM
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:33 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:34 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:35 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:36 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:37 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:38 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:39 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:40 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:41 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: ACCEPTED)
> 2019-12-12 15:11:42 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:42 INFO Client:54 -
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 10.16.15.59
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: UNDEFINED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> 2019-12-12 15:11:43 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:44 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:45 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:46 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:47 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:48 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:49 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:50 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:51 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:52 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:53 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:54 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:55 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:56 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:57 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:58 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:11:59 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:00 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:01 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:02 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:03 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:04 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:05 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:06 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:07 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:08 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: RUNNING)
> 2019-12-12 15:12:09 INFO Client:54 - Application report for
> application_1549925837808_10474661 (state: FINISHED)
> 2019-12-12 15:12:09 INFO Client:54 -
> client token: N/A
> diagnostics: User class threw exception: java.lang.RuntimeException: error
> execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause:
> Job aborted.
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
> Caused by: org.apache.spark.SparkException: Job aborted.
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
> at
> org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
> at
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task
> 0.3 in stage 1.0 (TID 5, host.dns.example.com, executor 15):
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/io/MetricsIOWrapper
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
> at
> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.MetricsIOWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 16 more
>
> Driver stacktrace:
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
> at scala.Option.foreach(Option.scala:257)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
> ... 16 more
> Caused by: org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/io/MetricsIOWrapper
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
> at
> org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
> at
> org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.MetricsIOWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 16 more
>
> ApplicationMaster host: 10.16.15.59
> ApplicationMaster RPC port: 0
> queue: root.hadoop
> start time: 1576134654497
> final status: FAILED
> tracking URL:
> http://host.example.com:8981/proxy/application_1549925837808_10474661/
> user: hadoop
> Exception in thread "main" org.apache.spark.SparkException: Application
> application_1549925837808_10474661 finished with failed status
> at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
> at
> org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409
> 2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory
> /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1
> The command is:
> export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop &&
> /usr/local/spark/bin/spark-submit --class
> org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40
> --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf
> spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
> spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec
> --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn
> --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf
> spark.hadoop.yarn.timeline-service.enabled=false --conf
> spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
> spark.eventLog.dir=hdfs:///kylin/spark-history --conf
> spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1
> --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false
> --conf spark.yarn.executor.memoryOverhead=1024 --conf
> spark.hadoop.dfs.replication=2 --conf
> spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G
> --conf spark.submit.deployMode=cluster --conf
> spark.shuffle.service.enabled=true --jars
> /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar,
> /usr/local/kylin/lib/kylin-job-2.6.4.jar -className
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile
> -counterOutput
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter
> -cubename test1 -output
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile
> -input
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/
> -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata
> -hbaseConfPath
> hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> This error message is very like the tips on the tutorial which I mentioned
> above. I've tried to put hbase-hadoop-compat-1.1.5.jar and
> hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I
> found there's no spark folder under the $KYLIN_HOME. This is strange
> because the tutorial said Kylin has a built-in Spark. Maybe the docs are
> out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put
> these two jars into the "jars" folder. But when after I resume the job,
> that error happened again. Nothing helps.
> I've also tried to put these two jars under the $SPARK_HOME/jars. But it
> still doesn't work.
>
>
>
> These versions are my current setup:
>
> Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz
>
> Spark: 2.3.2
>
> HBase: 1.1.5
>
> Hadoop: 2.7.2
>
>
>
> Thanks,
>
> Liang
>
>
>
>
>
Re: Build Cube with Spark Failed at Step: Convert Cuboid Data to
HFile
Posted by Wang rupeng <wa...@live.cn>.
Hi,
It seems like environment problem. What’s your cdh or hdp version?
-------------------
Best wishes,
Rupeng Wang
发件人: liang <pw...@gmail.com>
答复: <us...@kylin.apache.org>
日期: 2019年12月12日 星期四 15:46
收件人: <us...@kylin.apache.org>
主题: Build Cube with Spark Failed at Step: Convert Cuboid Data to HFile
Hi there,
I'm trying to build cube with Spark. I've followed the instructions in this tutorial: Build Cube with Spark<http://kylin.apache.org/docs/tutorial/cube_spark.html>. But the job failed at step: "Convert Cuboid Data to HFile". The logs shows "ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper"
org.apache.kylin.engine.spark.exception.SparkException: OS command error exit with return code: 1, error message: 2019-12-12 15:10:46 WARN SparkConf:66 - The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
2019-12-12 15:10:48 INFO RMProxy:98 - Connecting to ResourceManager at host/10.16.15.15:9032<http://10.16.15.15:9032>
2019-12-12 15:10:48 INFO Client:54 - Requesting a new application from cluster with 150 NodeManagers
2019-12-12 15:10:48 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (49150 MB per container)
2019-12-12 15:10:48 INFO Client:54 - Will allocate AM container, with 2432 MB memory including 384 MB overhead
2019-12-12 15:10:48 INFO Client:54 - Setting up container launch context for our AM
2019-12-12 15:10:48 INFO Client:54 - Setting up the launch environment for our AM container
2019-12-12 15:10:48 INFO Client:54 - Preparing resources for our AM container
2019-12-12 15:10:50 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://ns1/apps/spark2/spark2.3.2-libs.jar
2019-12-12 15:10:50 INFO Client:54 - Uploading resource file:/usr/local/kylin/lib/kylin-job-2.6.4.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/kylin-job-2.6.4.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-common-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-server-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-client-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-protocol-1.1.5.jar
2019-12-12 15:10:51 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop-compat-1.1.5.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hive-jdbc-1.2.1-standalone.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/htrace-core-3.1.0-incubating.jar
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/metrics-core-2.2.0.jar
2019-12-12 15:10:52 WARN Client:66 - Same path resource file:///usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar added multiple times to distributed cache.
2019-12-12 15:10:52 INFO Client:54 - Uploading resource file:/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/hbase-hadoop2-compat-1.1.5.jar
2019-12-12 15:10:53 INFO Client:54 - Uploading resource file:/tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409/__spark_conf__6376154711081920315.zip -> hdfs://ns1/user/hadoop/.sparkStaging/application_1549925837808_10474661/__spark_conf__.zip
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls to: hadoop
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls to: hadoop
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing view acls groups to:
2019-12-12 15:10:54 INFO SecurityManager:54 - Changing modify acls groups to:
2019-12-12 15:10:54 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
2019-12-12 15:10:54 INFO Client:54 - Submitting application application_1549925837808_10474661 to ResourceManager
2019-12-12 15:10:54 INFO YarnClientImpl:273 - Submitted application application_1549925837808_10474661
2019-12-12 15:10:55 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:55 INFO Client:54 -
client token: N/A
diagnostics: [Thu Dec 12 15:10:55 +0800 2019] Application is added to the scheduler and is not yet activated. (Resource request: <memory:3072, vCores:1> exceeds current queue or its parents maximum resource allowed).
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:10:56 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:57 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:58 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:10:59 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:00 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:01 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:02 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:03 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:04 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:05 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:06 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:07 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:08 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:09 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:10 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:11 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:11 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.16.15.244
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:12 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:13 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:14 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:15 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:16 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:17 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:18 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:19 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:20 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:21 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:22 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:23 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:24 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:25 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:26 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:27 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:28 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:29 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:30 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:31 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:32 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:32 INFO Client:54 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:33 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:34 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:35 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:36 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:37 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:38 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:39 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:40 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:41 INFO Client:54 - Application report for application_1549925837808_10474661 (state: ACCEPTED)
2019-12-12 15:11:42 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:42 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.16.15.59
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: UNDEFINED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
2019-12-12 15:11:43 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:44 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:45 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:46 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:47 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:48 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:49 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:50 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:51 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:52 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:53 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:54 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:55 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:56 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:57 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:58 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:11:59 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:00 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:01 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:02 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:03 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:04 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:05 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:06 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:07 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:08 INFO Client:54 - Application report for application_1549925837808_10474661 (state: RUNNING)
2019-12-12 15:12:09 INFO Client:54 - Application report for application_1549925837808_10474661 (state: FINISHED)
2019-12-12 15:12:09 INFO Client:54 -
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted.
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1083)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1081)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1081)
at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopDataset(JavaPairRDD.scala:831)
at org.apache.kylin.storage.hbase.steps.SparkCubeHFile.execute(SparkCubeHFile.java:238)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 6 more
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 5, host.dns.example.com<http://host.dns.example.com>, executor 15): org.apache.spark.SparkException: Task failed while writing rows
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/io/MetricsIOWrapper
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org<http://org.apache.spark.scheduler.DAGScheduler.org>$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2087)
at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
... 16 more
Caused by: org.apache.spark.SparkException: Task failed while writing rows
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/io/MetricsIOWrapper
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.MetricsIOWrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
ApplicationMaster host: 10.16.15.59
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1576134654497
final status: FAILED
tracking URL: http://host.example.com:8981/proxy/application_1549925837808_10474661/
user: hadoop
Exception in thread "main" org.apache.spark.SparkException: Application application_1549925837808_10474661 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Shutdown hook called
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-68d94435-5aa0-4560-8e1a-9fc9fbe00409
2019-12-12 15:12:09 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-b48e440b-4d74-4ecf-b500-38fe7cc542d1
The command is:
export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2//etc/hadoop && /usr/local/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry --conf spark.executor.instances=40 --conf spark.network.timeout=600 --conf spark.yarn.queue=default --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec --conf spark.dynamicAllocation.enabled=true --conf spark.master=yarn --conf spark.dynamicAllocation.executorIdleTimeout=300 --conf spark.hadoop.yarn.timeline-service.enabled=false --conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///kylin/spark-history --conf spark.dynamicAllocation.minExecutors=1 --conf spark.executor.cores=1 --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=false --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.hadoop.dfs.replication=2 --conf spark.dynamicAllocation.maxExecutors=1000 --conf spark.driver.memory=2G --conf spark.submit.deployMode=cluster --conf spark.shuffle.service.enabled=true --jars /usr/local/hbase-1.1.5/lib/hbase-common-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-server-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-client-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-protocol-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hive-1.2.1/lib/hive-jdbc-1.2.1-standalone.jar,/usr/local/hbase-1.1.5/lib/htrace-core-3.1.0-incubating.jar,/usr/local/hbase-1.1.5/lib/metrics-core-2.2.0.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop-compat-1.1.5.jar,/usr/local/hbase-1.1.5/lib/hbase-hadoop2-compat-1.1.5.jar, /usr/local/kylin/lib/kylin-job-2.6.4.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/rowkey_stats/part-r-00000_hfile -counterOutput hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/counter -cubename test1 -output hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/hfile -input hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/cuboid/ -segmentId 999b74f8-672a-37c4-6757-f86ef7db3d60 -metaUrl kylin_metadata@hdfs,path=hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/test1/metadata -hbaseConfPath hdfs://ns1/data/kylin/kylin_metadata/kylin-d80429c4-2b09-9b3b-2fda-329e0fa64b63/hbase-conf.xml
at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:347)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
This error message is very like the tips on the tutorial which I mentioned above. I've tried to put hbase-hadoop-compat-1.1.5.jar and hbase-hadoop2-compat-1.1.5.jar under the $KYLIN_HOME/spark/jars. But I found there's no spark folder under the $KYLIN_HOME. This is strange because the tutorial said Kylin has a built-in Spark. Maybe the docs are out-dated? I have to create $KYLIN_HOME/spark/jars folder manually and put these two jars into the "jars" folder. But when after I resume the job, that error happened again. Nothing helps.
I've also tried to put these two jars under the $SPARK_HOME/jars. But it still doesn't work.
These versions are my current setup:
Kylin: apache-kylin-2.6.4-bin-hbase1x.tar.gz
Spark: 2.3.2
HBase: 1.1.5
Hadoop: 2.7.2
Thanks,
Liang