You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 吕卓然 <lv...@fosun.com> on 2017/05/17 02:27:51 UTC

A problem in cube building with SPARK

Hi all,

Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info:

17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
                    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
                    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
                    at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
                    at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
                    at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
                    at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
                    at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
                    at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
                    at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
                    at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
                    at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
                    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
                    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
                    at org.apache.spark.scheduler.Task.run(Task.scala:89)
                    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

Any suggestions would help.

Thanks,
Zhuoran

Re: 答复: 答复: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Zuoran, did you check Spark history server to see what's wrong in the
executors? Is your kylin running it on a cluster and some nodes couldn't
connect with hbase, or the "hbase-site.xml" uses address that couldn't
visit from external nodes? Please do a check.

2017-05-17 13:40 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Sure, no problem.
>
> Here it is.
>
> OS command error exit with 1 -- export HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir
> && /data/kylin/kylin/spark/bin/spark-submit --class
> org.apache.kylin.common.util.SparkEntry  --conf
> spark.executor.instances=1  --conf spark.yarn.jar=hdfs://nsha/
> kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar  --conf
> spark.yarn.queue=default  --conf spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs
> --conf spark.master=yarn  --conf spark.executor.memory=4G  --conf
> spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///ahz/var/spark-logs
> --conf spark.executor.cores=2  --conf spark.submit.deployMode=cluster
> --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars
> /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.
> 6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/
> jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/
> parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-
> cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.
> p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/
> parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.
> 0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.
> p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-
> 5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, /data/kylin/kylin/lib/kylin-job-2.0.0.jar
> -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable
> default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6
> -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer
> -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6
> -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> Abstract Application args:-hiveTable default.kylin_intermediate_
> kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output
> hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> Using Spark's default log4j profile: org/apache/spark/log4j-
> defaults.properties
> 17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3
> 17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin
> 17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin
> 17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(kylin); users
> with modify permissions: Set(kylin)
> 17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver'
> on port 42715.
> 17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started
> 17/05/17 09:13:37 INFO Remoting: Starting remoting
> 17/05/17 09:13:38 INFO Utils: Successfully started service
> 'sparkDriverActorSystem' on port 39514.
> 17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriverActorSystem@10.24.26.191:39514]
> 17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker
> 17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster
> 17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at
> /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211
> 17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity
> 511.1 MB
> 17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator
> 17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
> 17/05/17 09:13:38 INFO SparkUI: Started SparkUI at
> http://10.24.26.191:4040
> 17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-
> 7d7509ef-786d-4fa4-a75e-bb3cb1015c77
> 17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server
> 17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file
> server' on port 34111.
> 17/05/17 09:13:38 INFO SparkContext: Added JAR
> file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at
> http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with
> timestamp 1494983618934
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at
> http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with
> timestamp 1494983618938
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618942
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618943
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618953
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at
> http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp
> 1494983618953
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at
> http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp
> 1494983618957
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar
> at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp
> 1494983618968
> 17/05/17 09:13:39 INFO Client: Requesting a new application from cluster
> with 4 NodeManagers
> 17/05/17 09:13:39 INFO Client: Verifying our application has not requested
> more than the maximum memory capability of the cluster (18852 MB per
> container)
> 17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB
> memory including 384 MB overhead
> 17/05/17 09:13:39 INFO Client: Setting up container launch context for our
> AM
> 17/05/17 09:13:39 INFO Client: Setting up the launch environment for our
> AM container
> 17/05/17 09:13:39 INFO Client: Preparing resources for our AM container
> 17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode:
> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568
> 17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549
> for kylin on ha-hdfs:nsha
> 17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI
> thrift://fonova-ahz-cdh03:9083
> 17/05/17 09:13:40 INFO metastore: Connected to metastore.
> 17/05/17 09:13:41 INFO Client: Source and destination file systems are the
> same. Not copying hdfs://nsha/kylin/spark/spark-
> assembly-1.6.3-hadoop2.6.0.jar
> 17/05/17 09:13:41 INFO Client: Uploading resource
> file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml ->
> hdfs://nsha/user/kylin/.sparkStaging/application_
> 1493867056374_0568/hbase-site.xml
> 17/05/17 09:13:41 INFO Client: Uploading resource
> file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip
> -> hdfs://nsha/user/kylin/.sparkStaging/application_
> 1493867056374_0568/__spark_conf__5160745721183215779.zip
> 17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin
> 17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin
> 17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(kylin); users
> with modify permissions: Set(kylin)
> 17/05/17 09:13:41 INFO Client: Submitting application 568 to
> ResourceManager
> 17/05/17 09:13:41 INFO YarnClientImpl: Submitted application
> application_1493867056374_0568
> 17/05/17 09:13:42 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:42 INFO Client:
>           client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
>          diagnostics: N/A
>          ApplicationMaster host: N/A
>          ApplicationMaster RPC port: -1
>          queue: root.users.kylin
>          start time: 1494983621477
>          final status: UNDEFINED
>          tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_
> 1493867056374_0568/
>          user: kylin
> 17/05/17 09:13:43 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:44 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:45 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as NettyRpcEndpointRef(null)
> 17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter.
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
> Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES ->
> http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http://
> fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568),
> /proxy/application_1493867056374_0568
> 17/05/17 09:13:45 INFO JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 17/05/17 09:13:46 INFO Client: Application report for
> application_1493867056374_0568 (state: RUNNING)
> 17/05/17 09:13:46 INFO Client:
>           client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
>          diagnostics: N/A
>          ApplicationMaster host: 10.28.30.122
>          ApplicationMaster RPC port: 0
>          queue: root.users.kylin
>          start time: 1494983621477
>          final status: UNDEFINED
>          tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_
> 1493867056374_0568/
>          user: kylin
> 17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application
> application_1493867056374_0568 has started running.
> 17/05/17 09:13:46 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037.
> 17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037
> 17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager
> 17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block
> manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver,
> 10.24.26.191, 44037)
> 17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager
> 17/05/17 09:13:46 INFO EventLoggingListener: Logging events to
> hdfs:///ahz/var/spark-logs/application_1493867056374_0568
> 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor
> NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1
> 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.8
> 17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to
> class path
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-tools-log4j.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-tools-log4j.properties at
> http://10.24.26.191:34111/files/kylin-tools-log4j.properties with
> timestamp 1494983631827
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_hive_conf.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml
> 17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block
> manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1,
> fonova-ahz-cdh34, 34665)
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/
> files/kylin_hive_conf.xml with timestamp 1494983631843
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-server-log4j.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-server-log4j.properties at
> http://10.24.26.191:34111/files/kylin-server-log4j.properties with
> timestamp 1494983631859
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/
> files/kylin.properties with timestamp 1494983631873
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/
> files/kylin_job_conf.xml with timestamp 1494983631884
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-kafka-consumer.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-kafka-consumer.xml at
> http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp
> 1494983631898
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf_inmem.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at
> http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp
> 1494983631904
> 17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf
> 17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from
> getInstanceFromEnv : 1577864993
> 17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version
> 1.2.1
> 17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
> 17/05/17 09:13:52 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims
> for Hadoop version 2.6.0
> 17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called
> 17/05/17 09:13:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown
> unknown - will be ignored
> 17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2
> unknown - will be ignored
> 17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes
> with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,
> Partition,Database,Type,FieldSchema,Order"
> 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB
> is DERBY
> 17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore
> 17/05/17 09:13:56 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording
> the schema version 1.2.0
> 17/05/17 09:13:57 WARN ObjectStore: Failed to get database default,
> returning NoSuchObjectException
> 17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore
> 17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore
> 17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role,
> since config is empty
> 17/05/17 09:13:57 INFO SessionState: Created local directory:
> /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources
> 17/05/17 09:13:57 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
> 17/05/17 09:13:57 INFO SessionState: Created local directory:
> /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
> 17/05/17 09:13:57 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db
> 17/05/17 09:13:57 INFO HiveContext: default warehouse location is
> /user/hive/warehouse
> 17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection
> version 1.2.1 using Spark classes.
> 17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
> 17/05/17 09:13:57 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims
> for Hadoop version 2.6.0
> 17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI
> thrift://fonova-ahz-cdh03:9083
> 17/05/17 09:13:58 INFO metastore: Connected to metastore.
> 17/05/17 09:13:58 INFO SessionState: Created local directory:
> /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources
> 17/05/17 09:13:58 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
> 17/05/17 09:13:58 INFO SessionState: Created local directory:
> /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
> 17/05/17 09:13:58 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db
> 17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config
> kylin_metadata@hbase
> 17/05/17 09:13:58 INFO ResourceStore: Using metadata url
> kylin_metadata@hbase for resource store
> 17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed,
> creating a new one
> 17/05/17 09:13:58 INFO RecoverableZooKeeper: Process
> identifier=hconnection-0x5d01a2eb connecting to ZooKeeper
> ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965,
> built on 02/20/2014 09:09 GMT
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova-
> app-gw01
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle
> Corporation
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/
> java/jdk1.8.0_91/jre
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/
> data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/
> spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/
> spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/
> spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/
> spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path=
> :/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/
> hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1.
> cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/
> usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA>
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0-
> 514.6.2.el7.x86_64
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/
> kylin/kylin/bin
> 17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection,
> connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
> sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0,
> quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181,
> baseZNode=/ahz/hbase
> 17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server
> fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to
> fonova-ahz-cdh04/10.47.54.56:2181, initiating session
> 17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on
> server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac,
> negotiated timeout = 60000
> 17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated.
> Instead, use io.native.lib.available
> 17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder
> kylin_metadata(key='/cube')@kylin_metadata@hbase
> 17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with
> config kylin_metadata@hbase
> 17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from
> folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
> 17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with
> metadata url kylin_metadata@hbase
> 17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types
> from kylin config
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.
> HLLCMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.
> bitmap.BitmapMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class
> org.apache.kylin.measure.topn.TopNMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class
> org.apache.kylin.measure.raw.RawMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn),
> class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$
> Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> PERCENTILE(percentile), class org.apache.kylin.measure.percentile.
> PercentileMeasureType$Factory
> 17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s)
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being
> CUBE[name=kylin_sales_cube] having 2 segments
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being
> CUBE[name=yuyuan_flow_test] having 1 segments
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark
> being CUBE[name=yuyuan_flow_test_spark] having 1 segments
> 17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes
> 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 95.7 KB, free 511.0 MB)
> 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 9.1 KB, free 511.0 MB)
> 17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB)
> 17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at
> SparkCubingByLayer.java:166
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 33.5 KB, free 511.0 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 3.5 KB, free 511.0 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at
> SparkCubingByLayer.java:167
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/
> NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/
> LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_
> USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_
> REGION/6e20015c-7249-4436-807e-b260361279dd.dict
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 96.1 KB, free 510.9 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as
> bytes in memory (estimated size 9.2 KB, free 510.9 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in
> memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at
> SparkCubingByLayer.java:170
> 17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on
> all cuboids) ? : true
> 17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in
> memory (estimated size 616.1 KB, free 510.3 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as
> bytes in memory (estimated size 46.3 KB, free 510.2 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in
> memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at
> SparkCubingByLayer.java:193
> 17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550
> for kylin on ha-hdfs:nsha
> 17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN
> token 1550 for kylin)
> 17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1
> 17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition
> number: 1
> 17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all
> cuboids are 262143, total size is 0.24426913261413574
> 17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated.
> Instead, use mapreduce.output.fileoutputformat.outputdir
> 17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551
> for kylin on ha-hdfs:nsha
> 17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN
> token 1551 for kylin)
> 17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile
> at SparkCubingByLayer.java:288
> 17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at
> SparkCubingByLayer.java:193)
> 17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at
> SparkCubingByLayer.java:288) with 1 output partitions
> 17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1
> (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288)
> 17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage:
> List(ShuffleMapStage 0)
> 17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage
> 0)
> 17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0
> (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which
> has no missing parents
> 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in
> memory (estimated size 59.9 KB, free 510.2 MB)
> 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as
> bytes in memory (estimated size 25.6 KB, free 510.2 MB)
> 17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in
> memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB)
> 17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at
> DAGScheduler.scala:1006
> 17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from
> ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at
> SparkCubingByLayer.java:193)
> 17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
> 17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in
> memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB)
> 17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in
> memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB)
> 17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
> 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID
> 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID
> 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4
> times; aborting job
> 17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0
> 17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at
> SparkCubingByLayer.java:193) failed in 199.860 s
> 17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile
> at SparkCubingByLayer.java:288, took 200.053479 s
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
>          at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:42)
>          at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
>          at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
>          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
>          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
> 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException:
> Failed to find metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> Driver stacktrace:
>          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
> scheduler$DAGScheduler$$failJobAndIndependentStages(
> DAGScheduler.scala:1431)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> abortStage$1.apply(DAGScheduler.scala:1419)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> abortStage$1.apply(DAGScheduler.scala:1418)
>          at scala.collection.mutable.ResizableArray$class.foreach(
> ResizableArray.scala:59)
>          at scala.collection.mutable.ArrayBuffer.foreach(
> ArrayBuffer.scala:47)
>          at org.apache.spark.scheduler.DAGScheduler.abortStage(
> DAGScheduler.scala:1418)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>          at scala.Option.foreach(Option.scala:236)
>          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(
> DAGScheduler.scala:799)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> doOnReceive(DAGScheduler.scala:1640)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1599)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1588)
>          at org.apache.spark.util.EventLoop$$anon$1.run(
> EventLoop.scala:48)
>          at org.apache.spark.scheduler.DAGScheduler.runJob(
> DAGScheduler.scala:620)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:150)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:111)
>          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
>          at org.apache.spark.rdd.PairRDDFunctions.
> saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:150)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:111)
>          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
>          at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(
> PairRDDFunctions.scala:985)
>          at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(
> JavaPairRDD.scala:800)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(
> SparkCubingByLayer.java:288)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:257)
>          at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
>          ... 10 more
> Caused by: java.lang.IllegalArgumentException: Failed to find metadata
> store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
> 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation:
> Closing master protocol: MasterService
> 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation:
> Closing zookeeper sessionid=0x25bd16992de5eac
> 17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook
> 17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed
> 17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down
> 17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at
> http://10.24.26.191:4040
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all
> executors
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor
> thread
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to
> shut down
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped
> 17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared
> 17/05/17 09:17:22 INFO BlockManager: BlockManager stopped
> 17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped
> 17/05/17 09:17:22 INFO OutputCommitCoordinator$
> OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
> 17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext
> 17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-
> 7d7509ef-786d-4fa4-a75e-bb3cb1015c77
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote
> daemon shut down; proceeding with flushing remote transports.
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc
>
>
>
> Thanks,
> Zhuoran
>
>
> 发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
> 发送时间: 2017年5月17日 13:36
> 收件人: dev
> 抄送: user@kylin.apache.org
> 主题: Re: 答复: A problem in cube building with SPARK
>
> The mailing list filters attachement. Could you please paste it directly,
> or share it with gist?
>
> 2017-05-17 12:17 GMT+08:00 吕卓然 <lvzhuoran@fosun.com<mailto:lv
> zhuoran@fosun.com>>:
> Hi Shaofeng,
>
> I’ve attached the error log in the attachment..
>
>
>
>
>
>
>
>
> 发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:
> shaofengshi@apache.org>]
> 发送时间: 2017年5月17日 10:38
> 收件人: user
> 抄送: dev@kylin.apache.org<ma...@kylin.apache.org>
> 主题: Re: A problem in cube building with SPARK
>
> Hi zhuoran, is there any more messages before this error? This error is
> not the root cause.
>
> 2017-05-17 10:27 GMT+08:00 吕卓然 <lvzhuoran@fosun.com<mailto:lv
> zhuoran@fosun.com>>:
> Hi all,
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>                     at java.lang.Thread.run(Thread.java:745)
>
> Any suggestions would help.
>
> Thanks,
> Zhuoran
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re: 答复: 答复: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Zuoran, did you check Spark history server to see what's wrong in the
executors? Is your kylin running it on a cluster and some nodes couldn't
connect with hbase, or the "hbase-site.xml" uses address that couldn't
visit from external nodes? Please do a check.

2017-05-17 13:40 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Sure, no problem.
>
> Here it is.
>
> OS command error exit with 1 -- export HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir
> && /data/kylin/kylin/spark/bin/spark-submit --class
> org.apache.kylin.common.util.SparkEntry  --conf
> spark.executor.instances=1  --conf spark.yarn.jar=hdfs://nsha/
> kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar  --conf
> spark.yarn.queue=default  --conf spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs
> --conf spark.master=yarn  --conf spark.executor.memory=4G  --conf
> spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///ahz/var/spark-logs
> --conf spark.executor.cores=2  --conf spark.submit.deployMode=cluster
> --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars
> /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.
> 6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/
> jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/
> parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-
> cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.
> p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/
> parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.
> 0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.
> p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-
> 5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, /data/kylin/kylin/lib/kylin-job-2.0.0.jar
> -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable
> default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6
> -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer
> -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6
> -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> Abstract Application args:-hiveTable default.kylin_intermediate_
> kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output
> hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-
> 4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6
> -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
> Using Spark's default log4j profile: org/apache/spark/log4j-
> defaults.properties
> 17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3
> 17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin
> 17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin
> 17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(kylin); users
> with modify permissions: Set(kylin)
> 17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver'
> on port 42715.
> 17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started
> 17/05/17 09:13:37 INFO Remoting: Starting remoting
> 17/05/17 09:13:38 INFO Utils: Successfully started service
> 'sparkDriverActorSystem' on port 39514.
> 17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriverActorSystem@10.24.26.191:39514]
> 17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker
> 17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster
> 17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at
> /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211
> 17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity
> 511.1 MB
> 17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator
> 17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
> 17/05/17 09:13:38 INFO SparkUI: Started SparkUI at
> http://10.24.26.191:4040
> 17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-
> 7d7509ef-786d-4fa4-a75e-bb3cb1015c77
> 17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server
> 17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file
> server' on port 34111.
> 17/05/17 09:13:38 INFO SparkContext: Added JAR
> file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at
> http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with
> timestamp 1494983618934
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at
> http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with
> timestamp 1494983618938
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618942
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618943
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at
> http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with
> timestamp 1494983618953
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at
> http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp
> 1494983618953
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/
> CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at
> http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp
> 1494983618957
> 17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar
> at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp
> 1494983618968
> 17/05/17 09:13:39 INFO Client: Requesting a new application from cluster
> with 4 NodeManagers
> 17/05/17 09:13:39 INFO Client: Verifying our application has not requested
> more than the maximum memory capability of the cluster (18852 MB per
> container)
> 17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB
> memory including 384 MB overhead
> 17/05/17 09:13:39 INFO Client: Setting up container launch context for our
> AM
> 17/05/17 09:13:39 INFO Client: Setting up the launch environment for our
> AM container
> 17/05/17 09:13:39 INFO Client: Preparing resources for our AM container
> 17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode:
> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568
> 17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549
> for kylin on ha-hdfs:nsha
> 17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI
> thrift://fonova-ahz-cdh03:9083
> 17/05/17 09:13:40 INFO metastore: Connected to metastore.
> 17/05/17 09:13:41 INFO Client: Source and destination file systems are the
> same. Not copying hdfs://nsha/kylin/spark/spark-
> assembly-1.6.3-hadoop2.6.0.jar
> 17/05/17 09:13:41 INFO Client: Uploading resource
> file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml ->
> hdfs://nsha/user/kylin/.sparkStaging/application_
> 1493867056374_0568/hbase-site.xml
> 17/05/17 09:13:41 INFO Client: Uploading resource
> file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip
> -> hdfs://nsha/user/kylin/.sparkStaging/application_
> 1493867056374_0568/__spark_conf__5160745721183215779.zip
> 17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin
> 17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin
> 17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(kylin); users
> with modify permissions: Set(kylin)
> 17/05/17 09:13:41 INFO Client: Submitting application 568 to
> ResourceManager
> 17/05/17 09:13:41 INFO YarnClientImpl: Submitted application
> application_1493867056374_0568
> 17/05/17 09:13:42 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:42 INFO Client:
>           client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
>          diagnostics: N/A
>          ApplicationMaster host: N/A
>          ApplicationMaster RPC port: -1
>          queue: root.users.kylin
>          start time: 1494983621477
>          final status: UNDEFINED
>          tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_
> 1493867056374_0568/
>          user: kylin
> 17/05/17 09:13:43 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:44 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:45 INFO Client: Application report for
> application_1493867056374_0568 (state: ACCEPTED)
> 17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:
> ApplicationMaster registered as NettyRpcEndpointRef(null)
> 17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter.
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,
> Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES ->
> http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http://
> fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568),
> /proxy/application_1493867056374_0568
> 17/05/17 09:13:45 INFO JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 17/05/17 09:13:46 INFO Client: Application report for
> application_1493867056374_0568 (state: RUNNING)
> 17/05/17 09:13:46 INFO Client:
>           client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
>          diagnostics: N/A
>          ApplicationMaster host: 10.28.30.122
>          ApplicationMaster RPC port: 0
>          queue: root.users.kylin
>          start time: 1494983621477
>          final status: UNDEFINED
>          tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_
> 1493867056374_0568/
>          user: kylin
> 17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application
> application_1493867056374_0568 has started running.
> 17/05/17 09:13:46 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037.
> 17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037
> 17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager
> 17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block
> manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver,
> 10.24.26.191, 44037)
> 17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager
> 17/05/17 09:13:46 INFO EventLoggingListener: Logging events to
> hdfs:///ahz/var/spark-logs/application_1493867056374_0568
> 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor
> NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1
> 17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.8
> 17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to
> class path
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-tools-log4j.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-tools-log4j.properties at
> http://10.24.26.191:34111/files/kylin-tools-log4j.properties with
> timestamp 1494983631827
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_hive_conf.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml
> 17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block
> manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1,
> fonova-ahz-cdh34, 34665)
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/
> files/kylin_hive_conf.xml with timestamp 1494983631843
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-server-log4j.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-server-log4j.properties at
> http://10.24.26.191:34111/files/kylin-server-log4j.properties with
> timestamp 1494983631859
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/
> files/kylin.properties with timestamp 1494983631873
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/
> files/kylin_job_conf.xml with timestamp 1494983631884
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-kafka-consumer.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin-kafka-consumer.xml at
> http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp
> 1494983631898
> 17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf_inmem.xml
> to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-
> c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml
> 17/05/17 09:13:51 INFO SparkContext: Added file
> /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at
> http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp
> 1494983631904
> 17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf
> 17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from
> getInstanceFromEnv : 1577864993
> 17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version
> 1.2.1
> 17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
> 17/05/17 09:13:52 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims
> for Hadoop version 2.6.0
> 17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called
> 17/05/17 09:13:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown
> unknown - will be ignored
> 17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2
> unknown - will be ignored
> 17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes
> with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,
> Partition,Database,Type,FieldSchema,Order"
> 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
> is tagged as "embedded-only" so does not have its own datastore table.
> 17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB
> is DERBY
> 17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore
> 17/05/17 09:13:56 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording
> the schema version 1.2.0
> 17/05/17 09:13:57 WARN ObjectStore: Failed to get database default,
> returning NoSuchObjectException
> 17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore
> 17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore
> 17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role,
> since config is empty
> 17/05/17 09:13:57 INFO SessionState: Created local directory:
> /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources
> 17/05/17 09:13:57 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
> 17/05/17 09:13:57 INFO SessionState: Created local directory:
> /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
> 17/05/17 09:13:57 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db
> 17/05/17 09:13:57 INFO HiveContext: default warehouse location is
> /user/hive/warehouse
> 17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection
> version 1.2.1 using Spark classes.
> 17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
> 17/05/17 09:13:57 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims
> for Hadoop version 2.6.0
> 17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI
> thrift://fonova-ahz-cdh03:9083
> 17/05/17 09:13:58 INFO metastore: Connected to metastore.
> 17/05/17 09:13:58 INFO SessionState: Created local directory:
> /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources
> 17/05/17 09:13:58 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
> 17/05/17 09:13:58 INFO SessionState: Created local directory:
> /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
> 17/05/17 09:13:58 INFO SessionState: Created HDFS directory:
> /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db
> 17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config
> kylin_metadata@hbase
> 17/05/17 09:13:58 INFO ResourceStore: Using metadata url
> kylin_metadata@hbase for resource store
> 17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed,
> creating a new one
> 17/05/17 09:13:58 INFO RecoverableZooKeeper: Process
> identifier=hconnection-0x5d01a2eb connecting to ZooKeeper
> ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965,
> built on 02/20/2014 09:09 GMT
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova-
> app-gw01
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle
> Corporation
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/
> java/jdk1.8.0_91/jre
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/
> data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/
> spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/
> spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/
> spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/
> spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path=
> :/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/
> hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1.
> cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/
> usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA>
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0-
> 514.6.2.el7.x86_64
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin
> 17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/
> kylin/kylin/bin
> 17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection,
> connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
> sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0,
> quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181,
> baseZNode=/ahz/hbase
> 17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server
> fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to
> fonova-ahz-cdh04/10.47.54.56:2181, initiating session
> 17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on
> server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac,
> negotiated timeout = 60000
> 17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated.
> Instead, use io.native.lib.available
> 17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder
> kylin_metadata(key='/cube')@kylin_metadata@hbase
> 17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with
> config kylin_metadata@hbase
> 17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from
> folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
> 17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with
> metadata url kylin_metadata@hbase
> 17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types
> from kylin config
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.
> HLLCMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.
> bitmap.BitmapMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class
> org.apache.kylin.measure.topn.TopNMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class
> org.apache.kylin.measure.raw.RawMeasureType$Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn),
> class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$
> Factory
> 17/05/17 09:14:00 INFO MeasureTypeFactory: registering
> PERCENTILE(percentile), class org.apache.kylin.measure.percentile.
> PercentileMeasureType$Factory
> 17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s)
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being
> CUBE[name=kylin_sales_cube] having 2 segments
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being
> CUBE[name=yuyuan_flow_test] having 1 segments
> 17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark
> being CUBE[name=yuyuan_flow_test_spark] having 1 segments
> 17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes
> 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 95.7 KB, free 511.0 MB)
> 17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 9.1 KB, free 511.0 MB)
> 17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB)
> 17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at
> SparkCubingByLayer.java:166
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 33.5 KB, free 511.0 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 3.5 KB, free 511.0 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at
> SparkCubingByLayer.java:167
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/
> ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/
> NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/
> LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_
> GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_
> USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict
> 17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424)
> loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_
> REGION/6e20015c-7249-4436-807e-b260361279dd.dict
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 96.1 KB, free 510.9 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as
> bytes in memory (estimated size 9.2 KB, free 510.9 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in
> memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at
> SparkCubingByLayer.java:170
> 17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on
> all cuboids) ? : true
> 17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in
> memory (estimated size 616.1 KB, free 510.3 MB)
> 17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as
> bytes in memory (estimated size 46.3 KB, free 510.2 MB)
> 17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in
> memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB)
> 17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at
> SparkCubingByLayer.java:193
> 17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550
> for kylin on ha-hdfs:nsha
> 17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN
> token 1550 for kylin)
> 17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1
> 17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition
> number: 1
> 17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all
> cuboids are 262143, total size is 0.24426913261413574
> 17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated.
> Instead, use mapreduce.output.fileoutputformat.outputdir
> 17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551
> for kylin on ha-hdfs:nsha
> 17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN
> token 1551 for kylin)
> 17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile
> at SparkCubingByLayer.java:288
> 17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at
> SparkCubingByLayer.java:193)
> 17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at
> SparkCubingByLayer.java:288) with 1 output partitions
> 17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1
> (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288)
> 17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage:
> List(ShuffleMapStage 0)
> 17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage
> 0)
> 17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0
> (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which
> has no missing parents
> 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in
> memory (estimated size 59.9 KB, free 510.2 MB)
> 17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as
> bytes in memory (estimated size 25.6 KB, free 510.2 MB)
> 17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in
> memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB)
> 17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at
> DAGScheduler.scala:1006
> 17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from
> ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at
> SparkCubingByLayer.java:193)
> 17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
> 17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
> 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in
> memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB)
> 17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in
> memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB)
> 17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
> 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID
> 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID
> 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
> 17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4
> times; aborting job
> 17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks
> have all completed, from pool
> 17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0
> 17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at
> SparkCubingByLayer.java:193) failed in 199.860 s
> 17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile
> at SparkCubingByLayer.java:288, took 200.053479 s
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
>          at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:42)
>          at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
>          at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
>          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
>          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
> 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException:
> Failed to find metadata store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> Driver stacktrace:
>          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$
> scheduler$DAGScheduler$$failJobAndIndependentStages(
> DAGScheduler.scala:1431)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> abortStage$1.apply(DAGScheduler.scala:1419)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> abortStage$1.apply(DAGScheduler.scala:1418)
>          at scala.collection.mutable.ResizableArray$class.foreach(
> ResizableArray.scala:59)
>          at scala.collection.mutable.ArrayBuffer.foreach(
> ArrayBuffer.scala:47)
>          at org.apache.spark.scheduler.DAGScheduler.abortStage(
> DAGScheduler.scala:1418)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>          at org.apache.spark.scheduler.DAGScheduler$$anonfun$
> handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
>          at scala.Option.foreach(Option.scala:236)
>          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(
> DAGScheduler.scala:799)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> doOnReceive(DAGScheduler.scala:1640)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1599)
>          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.
> onReceive(DAGScheduler.scala:1588)
>          at org.apache.spark.util.EventLoop$$anon$1.run(
> EventLoop.scala:48)
>          at org.apache.spark.scheduler.DAGScheduler.runJob(
> DAGScheduler.scala:620)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
>          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:150)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:111)
>          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
>          at org.apache.spark.rdd.PairRDDFunctions.
> saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
>          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$
> saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:150)
>          at org.apache.spark.rdd.RDDOperationScope$.withScope(
> RDDOperationScope.scala:111)
>          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
>          at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(
> PairRDDFunctions.scala:985)
>          at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(
> JavaPairRDD.scala:800)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(
> SparkCubingByLayer.java:288)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:257)
>          at org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
>          ... 10 more
> Caused by: java.lang.IllegalArgumentException: Failed to find metadata
> store by url: kylin_metadata@hbase
>          at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>          at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>          at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>          at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>          at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>          at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>          at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>          at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>          at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>          at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>          at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:205)
>          at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:193)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>          at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>          at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>          at org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>          at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>          at org.apache.spark.scheduler.Task.run(Task.scala:89)
>          at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
> 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation:
> Closing master protocol: MasterService
> 17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation:
> Closing zookeeper sessionid=0x25bd16992de5eac
> 17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook
> 17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed
> 17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down
> 17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at
> http://10.24.26.191:4040
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all
> executors
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor
> thread
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to
> shut down
> 17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped
> 17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint:
> MapOutputTrackerMasterEndpoint stopped!
> 17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared
> 17/05/17 09:17:22 INFO BlockManager: BlockManager stopped
> 17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped
> 17/05/17 09:17:22 INFO OutputCommitCoordinator$
> OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
> 17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext
> 17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-
> 7d7509ef-786d-4fa4-a75e-bb3cb1015c77
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote
> daemon shut down; proceeding with flushing remote transports.
> 17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> 17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc
>
>
>
> Thanks,
> Zhuoran
>
>
> 发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
> 发送时间: 2017年5月17日 13:36
> 收件人: dev
> 抄送: user@kylin.apache.org
> 主题: Re: 答复: A problem in cube building with SPARK
>
> The mailing list filters attachement. Could you please paste it directly,
> or share it with gist?
>
> 2017-05-17 12:17 GMT+08:00 吕卓然 <lvzhuoran@fosun.com<mailto:lv
> zhuoran@fosun.com>>:
> Hi Shaofeng,
>
> I’ve attached the error log in the attachment..
>
>
>
>
>
>
>
>
> 发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:
> shaofengshi@apache.org>]
> 发送时间: 2017年5月17日 10:38
> 收件人: user
> 抄送: dev@kylin.apache.org<ma...@kylin.apache.org>
> 主题: Re: A problem in cube building with SPARK
>
> Hi zhuoran, is there any more messages before this error? This error is
> not the root cause.
>
> 2017-05-17 10:27 GMT+08:00 吕卓然 <lvzhuoran@fosun.com<mailto:lv
> zhuoran@fosun.com>>:
> Hi all,
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>                     at java.lang.Thread.run(Thread.java:745)
>
> Any suggestions would help.
>
> Thanks,
> Zhuoran
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

答复: 答复: A problem in cube building with SPARK

Posted by 吕卓然 <lv...@fosun.com>.
Sure, no problem.

Here it is.

OS command error exit with 1 -- export HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir && /data/kylin/kylin/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=1  --conf spark.yarn.jar=hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar  --conf spark.yarn.queue=default  --conf spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs  --conf spark.master=yarn  --conf spark.executor.memory=4G  --conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///ahz/var/spark-logs  --conf spark.executor.cores=2  --conf spark.submit.deployMode=cluster --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, /data/kylin/kylin/lib/kylin-job-2.0.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
Abstract Application args:-hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3
17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin
17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin
17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin)
17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver' on port 42715.
17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started
17/05/17 09:13:37 INFO Remoting: Starting remoting
17/05/17 09:13:38 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 39514.
17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.24.26.191:39514]
17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker
17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster
17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211
17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator
17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/05/17 09:13:38 INFO SparkUI: Started SparkUI at http://10.24.26.191:4040
17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77
17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server
17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file server' on port 34111.
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with timestamp 1494983618934
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with timestamp 1494983618938
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with timestamp 1494983618942
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with timestamp 1494983618943
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with timestamp 1494983618953
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp 1494983618953
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp 1494983618957
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp 1494983618968
17/05/17 09:13:39 INFO Client: Requesting a new application from cluster with 4 NodeManagers
17/05/17 09:13:39 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (18852 MB per container)
17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/05/17 09:13:39 INFO Client: Setting up container launch context for our AM
17/05/17 09:13:39 INFO Client: Setting up the launch environment for our AM container
17/05/17 09:13:39 INFO Client: Preparing resources for our AM container
17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568
17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549 for kylin on ha-hdfs:nsha
17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083
17/05/17 09:13:40 INFO metastore: Connected to metastore.
17/05/17 09:13:41 INFO Client: Source and destination file systems are the same. Not copying hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
17/05/17 09:13:41 INFO Client: Uploading resource file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/hbase-site.xml
17/05/17 09:13:41 INFO Client: Uploading resource file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/__spark_conf__5160745721183215779.zip
17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin
17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin
17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin)
17/05/17 09:13:41 INFO Client: Submitting application 568 to ResourceManager
17/05/17 09:13:41 INFO YarnClientImpl: Submitted application application_1493867056374_0568
17/05/17 09:13:42 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:42 INFO Client:
          client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.users.kylin
         start time: 1494983621477
         final status: UNDEFINED
         tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/
         user: kylin
17/05/17 09:13:43 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:44 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:45 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES -> http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http://fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568), /proxy/application_1493867056374_0568
17/05/17 09:13:45 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/17 09:13:46 INFO Client: Application report for application_1493867056374_0568 (state: RUNNING)
17/05/17 09:13:46 INFO Client:
          client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: 10.28.30.122
         ApplicationMaster RPC port: 0
         queue: root.users.kylin
         start time: 1494983621477
         final status: UNDEFINED
         tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/
         user: kylin
17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application application_1493867056374_0568 has started running.
17/05/17 09:13:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037.
17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037
17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager
17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver, 10.24.26.191, 44037)
17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager
17/05/17 09:13:46 INFO EventLoggingListener: Logging events to hdfs:///ahz/var/spark-logs/application_1493867056374_0568
17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1
17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to class path
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-tools-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-tools-log4j.properties at http://10.24.26.191:34111/files/kylin-tools-log4j.properties with timestamp 1494983631827
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_hive_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml
17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1, fonova-ahz-cdh34, 34665)
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/files/kylin_hive_conf.xml with timestamp 1494983631843
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-server-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-server-log4j.properties at http://10.24.26.191:34111/files/kylin-server-log4j.properties with timestamp 1494983631859
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/files/kylin.properties with timestamp 1494983631873
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/files/kylin_job_conf.xml with timestamp 1494983631884
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-kafka-consumer.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-kafka-consumer.xml at http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp 1494983631898
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf_inmem.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp 1494983631904
17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf
17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from getInstanceFromEnv : 1577864993
17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version 1.2.1
17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
17/05/17 09:13:52 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called
17/05/17 09:13:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore
17/05/17 09:13:56 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/05/17 09:13:57 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore
17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore
17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role, since config is empty
17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources
17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db
17/05/17 09:13:57 INFO HiveContext: default warehouse location is /user/hive/warehouse
17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
17/05/17 09:13:57 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083
17/05/17 09:13:58 INFO metastore: Connected to metastore.
17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources
17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db
17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config kylin_metadata@hbase
17/05/17 09:13:58 INFO ResourceStore: Using metadata url kylin_metadata@hbase for resource store
17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed, creating a new one
17/05/17 09:13:58 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5d01a2eb connecting to ZooKeeper ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
17/05/17 09:13:58 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova-app-gw01
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_91/jre
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path=:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA>
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0-514.6.2.el7.x86_64
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/kylin/kylin/bin
17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection, connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0, quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181, baseZNode=/ahz/hbase
17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using SASL (unknown error)
17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to fonova-ahz-cdh04/10.47.54.56:2181, initiating session
17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac, negotiated timeout = 60000
17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder kylin_metadata(key='/cube')@kylin_metadata@hbase
17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hbase
17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@hbase
17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types from kylin config
17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering PERCENTILE(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s)
17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 2 segments
17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being CUBE[name=yuyuan_flow_test] having 1 segments
17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark being CUBE[name=yuyuan_flow_test_spark] having 1 segments
17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes
17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 95.7 KB, free 511.0 MB)
17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 9.1 KB, free 511.0 MB)
17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB)
17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at SparkCubingByLayer.java:166
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 33.5 KB, free 511.0 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.5 KB, free 511.0 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at SparkCubingByLayer.java:167
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_REGION/6e20015c-7249-4436-807e-b260361279dd.dict
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 96.1 KB, free 510.9 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.2 KB, free 510.9 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at SparkCubingByLayer.java:170
17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on all cuboids) ? : true
17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 616.1 KB, free 510.3 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 46.3 KB, free 510.2 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at SparkCubingByLayer.java:193
17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550 for kylin on ha-hdfs:nsha
17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1550 for kylin)
17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1
17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition number: 1
17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all cuboids are 262143, total size is 0.24426913261413574
17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551 for kylin on ha-hdfs:nsha
17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1551 for kylin)
17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288
17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at SparkCubingByLayer.java:193)
17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288) with 1 output partitions
17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288)
17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which has no missing parents
17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 59.9 KB, free 510.2 MB)
17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 25.6 KB, free 510.2 MB)
17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB)
17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193)
17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB)
17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB)
17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0
17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at SparkCubingByLayer.java:193) failed in 199.860 s
17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288, took 200.053479 s
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer
         at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
         at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
         at scala.Option.foreach(Option.scala:236)
         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
         at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
         at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
         at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
         at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:985)
         at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:800)
         at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(SparkCubingByLayer.java:288)
         at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:257)
         at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
         ... 10 more
Caused by: java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x25bd16992de5eac
17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook
17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed
17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down
17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at http://10.24.26.191:4040
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all executors
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor thread
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to shut down
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped
17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared
17/05/17 09:17:22 INFO BlockManager: BlockManager stopped
17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped
17/05/17 09:17:22 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext
17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc



Thanks,
Zhuoran


发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年5月17日 13:36
收件人: dev
抄送: user@kylin.apache.org
主题: Re: 答复: A problem in cube building with SPARK

The mailing list filters attachement. Could you please paste it directly, or share it with gist?

2017-05-17 12:17 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi Shaofeng,

I’ve attached the error log in the attachment..








发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
发送时间: 2017年5月17日 10:38
收件人: user
抄送: dev@kylin.apache.org<ma...@kylin.apache.org>
主题: Re: A problem in cube building with SPARK

Hi zhuoran, is there any more messages before this error? This error is not the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi all,

Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info:

17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
                    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
                    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
                    at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
                    at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
                    at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
                    at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
                    at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
                    at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
                    at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
                    at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
                    at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
                    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
                    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
                    at org.apache.spark.scheduler.Task.run(Task.scala:89)
                    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

Any suggestions would help.

Thanks,
Zhuoran



--
Best regards,

Shaofeng Shi 史少锋




--
Best regards,

Shaofeng Shi 史少锋

答复: 答复: A problem in cube building with SPARK

Posted by 吕卓然 <lv...@fosun.com>.
Sure, no problem.

Here it is.

OS command error exit with 1 -- export HADOOP_CONF_DIR=/data/kylin/kylin/hadoop-conf-dir && /data/kylin/kylin/spark/bin/spark-submit --class org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=1  --conf spark.yarn.jar=hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar  --conf spark.yarn.queue=default  --conf spark.history.fs.logDirectory=hdfs:///ahz/var/spark-logs  --conf spark.master=yarn  --conf spark.executor.memory=4G  --conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///ahz/var/spark-logs  --conf spark.executor.cores=2  --conf spark.submit.deployMode=cluster --files /etc/hbase/conf.cloudera.hbase/hbase-site.xml --jars /data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar,/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar, /data/kylin/kylin/lib/kylin-job-2.0.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
SparkEntry args:-className org.apache.kylin.engine.spark.SparkCubingByLayer -hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
Abstract Application args:-hiveTable default.kylin_intermediate_kylin_sales_cube_2ffd651b_cdd9_469f_a957_67aee260abd6 -output hdfs:///kylin/kylin_metadata/kylin-fef41f82-b453-44d2-bc07-4d9be0a1f4e0/kylin_sales_cube/cuboid/ -segmentId 2ffd651b-cdd9-469f-a957-67aee260abd6 -confPath /data/kylin/kylin/conf -cubename kylin_sales_cube
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/05/17 09:13:36 INFO SparkContext: Running Spark version 1.6.3
17/05/17 09:13:37 INFO SecurityManager: Changing view acls to: kylin
17/05/17 09:13:37 INFO SecurityManager: Changing modify acls to: kylin
17/05/17 09:13:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin)
17/05/17 09:13:37 INFO Utils: Successfully started service 'sparkDriver' on port 42715.
17/05/17 09:13:37 INFO Slf4jLogger: Slf4jLogger started
17/05/17 09:13:37 INFO Remoting: Starting remoting
17/05/17 09:13:38 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 39514.
17/05/17 09:13:38 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.24.26.191:39514]
17/05/17 09:13:38 INFO SparkEnv: Registering MapOutputTracker
17/05/17 09:13:38 INFO SparkEnv: Registering BlockManagerMaster
17/05/17 09:13:38 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-6a57441d-d256-4ade-acf0-af6985732211
17/05/17 09:13:38 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/05/17 09:13:38 INFO SparkEnv: Registering OutputCommitCoordinator
17/05/17 09:13:38 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/05/17 09:13:38 INFO SparkUI: Started SparkUI at http://10.24.26.191:4040
17/05/17 09:13:38 INFO HttpFileServer: HTTP File server directory is /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77
17/05/17 09:13:38 INFO HttpServer: Starting HTTP Server
17/05/17 09:13:38 INFO Utils: Successfully started service 'HTTP file server' on port 34111.
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar at http://10.24.26.191:34111/jars/spark-assembly-1.6.3-hadoop2.6.0.jar with timestamp 1494983618934
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/htrace-core-3.2.0-incubating.jar at http://10.24.26.191:34111/jars/htrace-core-3.2.0-incubating.jar with timestamp 1494983618938
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-client-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-client-1.2.0-cdh5.8.4.jar with timestamp 1494983618942
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-common-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-common-1.2.0-cdh5.8.4.jar with timestamp 1494983618943
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/hbase-protocol-1.2.0-cdh5.8.4.jar at http://10.24.26.191:34111/jars/hbase-protocol-1.2.0-cdh5.8.4.jar with timestamp 1494983618953
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/metrics-core-2.2.0.jar at http://10.24.26.191:34111/jars/metrics-core-2.2.0.jar with timestamp 1494983618953
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/jars/guava-12.0.1.jar at http://10.24.26.191:34111/jars/guava-12.0.1.jar with timestamp 1494983618957
17/05/17 09:13:38 INFO SparkContext: Added JAR file:/data/kylin/kylin/lib/kylin-job-2.0.0.jar at http://10.24.26.191:34111/jars/kylin-job-2.0.0.jar with timestamp 1494983618968
17/05/17 09:13:39 INFO Client: Requesting a new application from cluster with 4 NodeManagers
17/05/17 09:13:39 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (18852 MB per container)
17/05/17 09:13:39 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
17/05/17 09:13:39 INFO Client: Setting up container launch context for our AM
17/05/17 09:13:39 INFO Client: Setting up the launch environment for our AM container
17/05/17 09:13:39 INFO Client: Preparing resources for our AM container
17/05/17 09:13:39 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568
17/05/17 09:13:40 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1549 for kylin on ha-hdfs:nsha
17/05/17 09:13:40 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083
17/05/17 09:13:40 INFO metastore: Connected to metastore.
17/05/17 09:13:41 INFO Client: Source and destination file systems are the same. Not copying hdfs://nsha/kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar
17/05/17 09:13:41 INFO Client: Uploading resource file:/etc/hbase/conf.cloudera.hbase/hbase-site.xml -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/hbase-site.xml
17/05/17 09:13:41 INFO Client: Uploading resource file:/tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/__spark_conf__5160745721183215779.zip -> hdfs://nsha/user/kylin/.sparkStaging/application_1493867056374_0568/__spark_conf__5160745721183215779.zip
17/05/17 09:13:41 INFO SecurityManager: Changing view acls to: kylin
17/05/17 09:13:41 INFO SecurityManager: Changing modify acls to: kylin
17/05/17 09:13:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kylin); users with modify permissions: Set(kylin)
17/05/17 09:13:41 INFO Client: Submitting application 568 to ResourceManager
17/05/17 09:13:41 INFO YarnClientImpl: Submitted application application_1493867056374_0568
17/05/17 09:13:42 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:42 INFO Client:
          client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: root.users.kylin
         start time: 1494983621477
         final status: UNDEFINED
         tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/
         user: kylin
17/05/17 09:13:43 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:44 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:45 INFO Client: Application report for application_1493867056374_0568 (state: ACCEPTED)
17/05/17 09:13:45 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/05/17 09:13:45 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> fonova-ahz-cdh03,fonova-ahz-cdh04, PROXY_URI_BASES -> http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568,http://fonova-ahz-cdh04:8088/proxy/application_1493867056374_0568), /proxy/application_1493867056374_0568
17/05/17 09:13:45 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/17 09:13:46 INFO Client: Application report for application_1493867056374_0568 (state: RUNNING)
17/05/17 09:13:46 INFO Client:
          client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: 10.28.30.122
         ApplicationMaster RPC port: 0
         queue: root.users.kylin
         start time: 1494983621477
         final status: UNDEFINED
         tracking URL: http://fonova-ahz-cdh03:8088/proxy/application_1493867056374_0568/
         user: kylin
17/05/17 09:13:46 INFO YarnClientSchedulerBackend: Application application_1493867056374_0568 has started running.
17/05/17 09:13:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44037.
17/05/17 09:13:46 INFO NettyBlockTransferService: Server created on 44037
17/05/17 09:13:46 INFO BlockManagerMaster: Trying to register BlockManager
17/05/17 09:13:46 INFO BlockManagerMasterEndpoint: Registering block manager 10.24.26.191:44037 with 511.1 MB RAM, BlockManagerId(driver, 10.24.26.191, 44037)
17/05/17 09:13:46 INFO BlockManagerMaster: Registered BlockManager
17/05/17 09:13:46 INFO EventLoggingListener: Logging events to hdfs:///ahz/var/spark-logs/application_1493867056374_0568
17/05/17 09:13:51 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (fonova-ahz-cdh34:36522) with ID 1
17/05/17 09:13:51 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/17 09:13:51 INFO ClassUtil: Adding path /data/kylin/kylin/conf to class path
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-tools-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-tools-log4j.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-tools-log4j.properties at http://10.24.26.191:34111/files/kylin-tools-log4j.properties with timestamp 1494983631827
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_hive_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_hive_conf.xml
17/05/17 09:13:51 INFO BlockManagerMasterEndpoint: Registering block manager fonova-ahz-cdh34:34665 with 2.7 GB RAM, BlockManagerId(1, fonova-ahz-cdh34, 34665)
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_hive_conf.xml at http://10.24.26.191:34111/files/kylin_hive_conf.xml with timestamp 1494983631843
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-server-log4j.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-server-log4j.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-server-log4j.properties at http://10.24.26.191:34111/files/kylin-server-log4j.properties with timestamp 1494983631859
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin.properties to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin.properties
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin.properties at http://10.24.26.191:34111/files/kylin.properties with timestamp 1494983631873
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf.xml at http://10.24.26.191:34111/files/kylin_job_conf.xml with timestamp 1494983631884
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin-kafka-consumer.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin-kafka-consumer.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin-kafka-consumer.xml at http://10.24.26.191:34111/files/kylin-kafka-consumer.xml with timestamp 1494983631898
17/05/17 09:13:51 INFO Utils: Copying /data/kylin/kylin/conf/kylin_job_conf_inmem.xml to /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/userFiles-c691c4b5-c867-4678-9b68-feaf9fce4277/kylin_job_conf_inmem.xml
17/05/17 09:13:51 INFO SparkContext: Added file /data/kylin/kylin/conf/kylin_job_conf_inmem.xml at http://10.24.26.191:34111/files/kylin_job_conf_inmem.xml with timestamp 1494983631904
17/05/17 09:13:51 INFO KylinConfig: Use KYLIN_CONF=/data/kylin/kylin/conf
17/05/17 09:13:51 INFO KylinConfig: Initialized a new KylinConfig from getInstanceFromEnv : 1577864993
17/05/17 09:13:52 INFO HiveContext: Initializing execution hive, version 1.2.1
17/05/17 09:13:52 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
17/05/17 09:13:52 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
17/05/17 09:13:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
17/05/17 09:13:52 INFO ObjectStore: ObjectStore, initialize called
17/05/17 09:13:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
17/05/17 09:13:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
17/05/17 09:13:54 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/05/17 09:13:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
17/05/17 09:13:56 INFO ObjectStore: Initialized ObjectStore
17/05/17 09:13:56 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/05/17 09:13:57 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/05/17 09:13:57 INFO HiveMetaStore: Added admin role in metastore
17/05/17 09:13:57 INFO HiveMetaStore: Added public role in metastore
17/05/17 09:13:57 INFO HiveMetaStore: No user is added in admin role, since config is empty
17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/4878224d-23a8-4f7e-adb2-0c261830abe1_resources
17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
17/05/17 09:13:57 INFO SessionState: Created local directory: /tmp/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1
17/05/17 09:13:57 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/4878224d-23a8-4f7e-adb2-0c261830abe1/_tmp_space.db
17/05/17 09:13:57 INFO HiveContext: default warehouse location is /user/hive/warehouse
17/05/17 09:13:57 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
17/05/17 09:13:57 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
17/05/17 09:13:57 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
17/05/17 09:13:58 INFO metastore: Trying to connect to metastore with URI thrift://fonova-ahz-cdh03:9083
17/05/17 09:13:58 INFO metastore: Connected to metastore.
17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/30eaa1b9-5883-42b8-9b8b-ac844590cac1_resources
17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
17/05/17 09:13:58 INFO SessionState: Created local directory: /tmp/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1
17/05/17 09:13:58 INFO SessionState: Created HDFS directory: /tmp/hive/kylin/30eaa1b9-5883-42b8-9b8b-ac844590cac1/_tmp_space.db
17/05/17 09:13:58 INFO CubeManager: Initializing CubeManager with config kylin_metadata@hbase
17/05/17 09:13:58 INFO ResourceStore: Using metadata url kylin_metadata@hbase for resource store
17/05/17 09:13:58 INFO HBaseConnection: connection is null or closed, creating a new one
17/05/17 09:13:58 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5d01a2eb connecting to ZooKeeper ensemble=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181
17/05/17 09:13:58 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
17/05/17 09:13:58 INFO ZooKeeper: Client environment:host.name=fonova-app-gw01
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.version=1.8.0_91
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_91/jre
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.class.path=/data/kylin/kylin/spark/conf/:/data/kylin/kylin/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar:/data/kylin/kylin/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/kylin/kylin/spark/lib/datanucleus-core-3.2.10.jar:/data/kylin/kylin/spark/lib/datanucleus-rdbms-3.2.9.jar:/data/kylin/kylin/hadoop-conf-dir/
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.library.path=:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
17/05/17 09:13:58 INFO ZooKeeper: Client environment:java.compiler=<NA>
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.name=Linux
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.arch=amd64
17/05/17 09:13:58 INFO ZooKeeper: Client environment:os.version=3.10.0-514.6.2.el7.x86_64
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.name=kylin
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.home=/data/kylin
17/05/17 09:13:58 INFO ZooKeeper: Client environment:user.dir=/data/kylin/kylin/bin
17/05/17 09:13:58 INFO ZooKeeper: Initiating client connection, connectString=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181 sessionTimeout=60000 watcher=hconnection-0x5d01a2eb0x0, quorum=fonova-ahz-cdh05:2181,fonova-ahz-cdh04:2181,fonova-ahz-cdh03:2181, baseZNode=/ahz/hbase
17/05/17 09:13:58 INFO ClientCnxn: Opening socket connection to server fonova-ahz-cdh04/10.47.54.56:2181. Will not attempt to authenticate using SASL (unknown error)
17/05/17 09:13:58 INFO ClientCnxn: Socket connection established to fonova-ahz-cdh04/10.47.54.56:2181, initiating session
17/05/17 09:13:58 INFO ClientCnxn: Session establishment complete on server fonova-ahz-cdh04/10.47.54.56:2181, sessionid = 0x25bd16992de5eac, negotiated timeout = 60000
17/05/17 09:13:59 INFO deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
17/05/17 09:13:59 INFO CubeManager: Loading Cube from folder kylin_metadata(key='/cube')@kylin_metadata@hbase
17/05/17 09:14:00 INFO CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hbase
17/05/17 09:14:00 INFO CubeDescManager: Reloading Cube Metadata from folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
17/05/17 09:14:00 INFO ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@hbase
17/05/17 09:14:00 INFO MeasureTypeFactory: Checking custom measure types from kylin config
17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
17/05/17 09:14:00 INFO MeasureTypeFactory: registering PERCENTILE(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
17/05/17 09:14:00 INFO CubeDescManager: Loaded 3 Cube(s)
17/05/17 09:14:00 INFO CubeManager: Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 2 segments
17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test being CUBE[name=yuyuan_flow_test] having 1 segments
17/05/17 09:14:00 INFO CubeManager: Reloaded cube yuyuan_flow_test_spark being CUBE[name=yuyuan_flow_test_spark] having 1 segments
17/05/17 09:14:00 INFO CubeManager: Loaded 3 cubes, fail on 0 cubes
17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 95.7 KB, free 511.0 MB)
17/05/17 09:14:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 9.1 KB, free 511.0 MB)
17/05/17 09:14:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.24.26.191:44037 (size: 9.1 KB, free: 511.1 MB)
17/05/17 09:14:00 INFO SparkContext: Created broadcast 0 from broadcast at SparkCubingByLayer.java:166
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 33.5 KB, free 511.0 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.5 KB, free 511.0 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.24.26.191:44037 (size: 3.5 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 1 from broadcast at SparkCubingByLayer.java:167
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/LEAF_CATEG_ID/3b10e09c-0ee7-48db-9f83-1f21552e0113.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/META_CATEG_NAME/579dd544-fbca-4595-a13d-b76acec741ff.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/669d36f7-bb09-4a0e-8239-84efd806ec0d.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/a89ec9e5-edef-4c1a-b50f-09a6f768b3f2.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_BUYER_LEVEL/61bd0c4f-bd22-4f38-99ce-1ffe5fa779d3.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_SELLER_LEVEL/e4f67b28-afb9-4dca-9989-c1393424a0da.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_COUNTRY/97c0a5cb-31d1-47fc-80ee-b9b07379ea1a.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_COUNTRY/NAME/9b410ddc-9004-4253-b39c-630954c0c4d0.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/LSTG_FORMAT_NAME/77b68501-3dce-4f7d-b860-652d2a8c2daa.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/SITE_ID/caf25161-7de0-4f8c-8a4c-1a91c92977fc.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_USER_ID/5f3ef593-0157-46d6-a700-e609f4ec7733.dict
17/05/17 09:14:01 INFO DictionaryManager: DictionaryManager(1544982424) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.KYLIN_SALES/OPS_REGION/6e20015c-7249-4436-807e-b260361279dd.dict
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 96.1 KB, free 510.9 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.2 KB, free 510.9 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.24.26.191:44037 (size: 9.2 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 2 from broadcast at SparkCubingByLayer.java:170
17/05/17 09:14:01 INFO SparkCubingByLayer: All measure are normal (agg on all cuboids) ? : true
17/05/17 09:14:01 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 616.1 KB, free 510.3 MB)
17/05/17 09:14:01 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 46.3 KB, free 510.2 MB)
17/05/17 09:14:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.24.26.191:44037 (size: 46.3 KB, free: 511.1 MB)
17/05/17 09:14:01 INFO SparkContext: Created broadcast 3 from javaRDD at SparkCubingByLayer.java:193
17/05/17 09:14:01 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1550 for kylin on ha-hdfs:nsha
17/05/17 09:14:01 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1550 for kylin)
17/05/17 09:14:01 INFO FileInputFormat: Total input paths to process : 1
17/05/17 09:14:01 INFO SparkCubingByLayer: encodedBaseRDD partition number: 1
17/05/17 09:14:01 INFO CubeStatsReader: Estimating size for layer 0, all cuboids are 262143, total size is 0.24426913261413574
17/05/17 09:14:01 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
17/05/17 09:14:02 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1551 for kylin on ha-hdfs:nsha
17/05/17 09:14:02 INFO TokenCache: Got dt for hdfs://nsha; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nsha, Ident: (HDFS_DELEGATION_TOKEN token 1551 for kylin)
17/05/17 09:14:02 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288
17/05/17 09:14:02 INFO DAGScheduler: Registering RDD 4 (mapToPair at SparkCubingByLayer.java:193)
17/05/17 09:14:02 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288) with 1 output partitions
17/05/17 09:14:02 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288)
17/05/17 09:14:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
17/05/17 09:14:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
17/05/17 09:14:02 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193), which has no missing parents
17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 59.9 KB, free 510.2 MB)
17/05/17 09:14:02 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 25.6 KB, free 510.2 MB)
17/05/17 09:14:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.24.26.191:44037 (size: 25.6 KB, free: 511.0 MB)
17/05/17 09:14:02 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/05/17 09:14:02 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[4] at mapToPair at SparkCubingByLayer.java:193)
17/05/17 09:14:02 INFO YarnScheduler: Adding task set 0.0 with 1 tasks
17/05/17 09:14:02 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:14:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on fonova-ahz-cdh34:34665 (size: 25.6 KB, free: 2.7 GB)
17/05/17 09:14:07 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on fonova-ahz-cdh34:34665 (size: 46.3 KB, free: 2.7 GB)
17/05/17 09:14:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:14:57 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:15:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:15:45 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:16:33 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:16:33 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34, partition 0,NODE_LOCAL, 3249 bytes)
17/05/17 09:17:22 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

17/05/17 09:17:22 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
17/05/17 09:17:22 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/05/17 09:17:22 INFO YarnScheduler: Cancelling stage 0
17/05/17 09:17:22 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at SparkCubingByLayer.java:193) failed in 199.860 s
17/05/17 09:17:22 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile at SparkCubingByLayer.java:288, took 200.053479 s
Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer
         at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
         at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
         at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
         at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
         at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
         at scala.Option.foreach(Option.scala:236)
         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
         at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1144)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
         at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
         at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1074)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:994)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
         at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:985)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
         at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
         at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:985)
         at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:800)
         at org.apache.kylin.engine.spark.SparkCubingByLayer.saveToHDFS(SparkCubingByLayer.java:288)
         at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:257)
         at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
         ... 10 more
Caused by: java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
         at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
         at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
         at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
         at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
         at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
         at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
         at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
         at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
         at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
         at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
         at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
         at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
         at org.apache.spark.scheduler.Task.run(Task.scala:89)
         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
         at java.lang.Thread.run(Thread.java:745)
17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/17 09:17:22 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x25bd16992de5eac
17/05/17 09:17:22 INFO SparkContext: Invoking stop() from shutdown hook
17/05/17 09:17:22 INFO ZooKeeper: Session: 0x25bd16992de5eac closed
17/05/17 09:17:22 INFO ClientCnxn: EventThread shut down
17/05/17 09:17:22 INFO SparkUI: Stopped Spark web UI at http://10.24.26.191:4040
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Shutting down all executors
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Interrupting monitor thread
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Asking each executor to shut down
17/05/17 09:17:22 INFO YarnClientSchedulerBackend: Stopped
17/05/17 09:17:22 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/05/17 09:17:22 INFO MemoryStore: MemoryStore cleared
17/05/17 09:17:22 INFO BlockManager: BlockManager stopped
17/05/17 09:17:22 INFO BlockManagerMaster: BlockManagerMaster stopped
17/05/17 09:17:22 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/05/17 09:17:22 INFO SparkContext: Successfully stopped SparkContext
17/05/17 09:17:22 INFO ShutdownHookManager: Shutdown hook called
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-3e8cd18d-d4d0-4180-9c29-18d5814b8439
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc/httpd-7d7509ef-786d-4fa4-a75e-bb3cb1015c77
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/05/17 09:17:22 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
17/05/17 09:17:22 INFO ShutdownHookManager: Deleting directory /tmp/spark-75ec3149-ccf1-4159-90a2-772bed73d3bc



Thanks,
Zhuoran


发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年5月17日 13:36
收件人: dev
抄送: user@kylin.apache.org
主题: Re: 答复: A problem in cube building with SPARK

The mailing list filters attachement. Could you please paste it directly, or share it with gist?

2017-05-17 12:17 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi Shaofeng,

I’ve attached the error log in the attachment..








发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
发送时间: 2017年5月17日 10:38
收件人: user
抄送: dev@kylin.apache.org<ma...@kylin.apache.org>
主题: Re: A problem in cube building with SPARK

Hi zhuoran, is there any more messages before this error? This error is not the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi all,

Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info:

17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
                    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
                    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
                    at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
                    at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
                    at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
                    at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
                    at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
                    at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
                    at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
                    at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
                    at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
                    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
                    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
                    at org.apache.spark.scheduler.Task.run(Task.scala:89)
                    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

Any suggestions would help.

Thanks,
Zhuoran



--
Best regards,

Shaofeng Shi 史少锋




--
Best regards,

Shaofeng Shi 史少锋

Re: 答复: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
The mailing list filters attachement. Could you please paste it directly,
or share it with gist?

2017-05-17 12:17 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Hi Shaofeng,
>
>
>
> I’ve attached the error log in the attachment..
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *发件人:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *发送时间:* 2017年5月17日 10:38
> *收件人:* user
> *抄送:* dev@kylin.apache.org
> *主题:* Re: A problem in cube building with SPARK
>
>
>
> Hi zhuoran, is there any more messages before this error? This error is
> not the root cause.
>
>
>
> 2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>:
>
> Hi all,
>
>
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
>
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>                     at java.lang.Thread.run(Thread.java:745)
>
>
>
> Any suggestions would help.
>
>
>
> Thanks,
>
> Zhuoran
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: 答复: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
The mailing list filters attachement. Could you please paste it directly,
or share it with gist?

2017-05-17 12:17 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Hi Shaofeng,
>
>
>
> I’ve attached the error log in the attachment..
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *发件人:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *发送时间:* 2017年5月17日 10:38
> *收件人:* user
> *抄送:* dev@kylin.apache.org
> *主题:* Re: A problem in cube building with SPARK
>
>
>
> Hi zhuoran, is there any more messages before this error? This error is
> not the root cause.
>
>
>
> 2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>:
>
> Hi all,
>
>
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
>
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>                     at java.lang.Thread.run(Thread.java:745)
>
>
>
> Any suggestions would help.
>
>
>
> Thanks,
>
> Zhuoran
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

答复: A problem in cube building with SPARK

Posted by 吕卓然 <lv...@fosun.com>.
Hi Shaofeng,

I’ve attached the error log in the attachment..








发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年5月17日 10:38
收件人: user
抄送: dev@kylin.apache.org
主题: Re: A problem in cube building with SPARK

Hi zhuoran, is there any more messages before this error? This error is not the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi all,

Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info:

17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
                    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
                    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
                    at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
                    at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
                    at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
                    at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
                    at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
                    at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
                    at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
                    at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
                    at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
                    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
                    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
                    at org.apache.spark.scheduler.Task.run(Task.scala:89)
                    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

Any suggestions would help.

Thanks,
Zhuoran



--
Best regards,

Shaofeng Shi 史少锋

答复: A problem in cube building with SPARK

Posted by 吕卓然 <lv...@fosun.com>.
Hi Shaofeng,

I’ve attached the error log in the attachment..








发件人: ShaoFeng Shi [mailto:shaofengshi@apache.org]
发送时间: 2017年5月17日 10:38
收件人: user
抄送: dev@kylin.apache.org
主题: Re: A problem in cube building with SPARK

Hi zhuoran, is there any more messages before this error? This error is not the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>>:
Hi all,

Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use MapReduce engine. However, when I try to use spark engine to build cube, it fails at step 7: Build Cube with Spark. Here is the log info:

17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
                    at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
                    at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
                    at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
                    at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
                    at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
                    at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
                    at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
                    at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
                    at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
                    at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
                    at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
                    at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
                    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
                    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
                    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
                    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
                    at org.apache.spark.scheduler.Task.run(Task.scala:89)
                    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:745)

Any suggestions would help.

Thanks,
Zhuoran



--
Best regards,

Shaofeng Shi 史少锋

Re: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi zhuoran, is there any more messages before this error? This error is not
the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Hi all,
>
>
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
>
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>                     at java.lang.Thread.run(Thread.java:745)
>
>
>
> Any suggestions would help.
>
>
>
> Thanks,
>
> Zhuoran
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: A problem in cube building with SPARK

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi zhuoran, is there any more messages before this error? This error is not
the root cause.

2017-05-17 10:27 GMT+08:00 吕卓然 <lv...@fosun.com>:

> Hi all,
>
>
>
> Currently I am using Kylin2.0.0 with CDH 5.8. It works fine when I use
> MapReduce engine. However, when I try to use spark engine to build cube, it
> fails at step 7: Build Cube with Spark. Here is the log info:
>
>
>
> 17/05/16 17:50:01 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
> fonova-ahz-cdh34): java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> createResourceStore(ResourceStore.java:99)
>
>                     at org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>
>                     at org.apache.kylin.cube.CubeDescManager.getStore(
> CubeDescManager.java:370)
>
>                     at org.apache.kylin.cube.CubeDescManager.
> reloadAllCubeDesc(CubeDescManager.java:298)
>
>                     at org.apache.kylin.cube.CubeDescManager.<init>(
> CubeDescManager.java:109)
>
>                     at org.apache.kylin.cube.CubeDescManager.getInstance(
> CubeDescManager.java:81)
>
>                     at org.apache.kylin.cube.CubeInstance.getDescriptor(
> CubeInstance.java:109)
>
>                     at org.apache.kylin.cube.CubeSegment.getCubeDesc(
> CubeSegment.java:119)
>
>                     at org.apache.kylin.cube.CubeSegment.isEnableSharding(
> CubeSegment.java:467)
>
>                     at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(
> RowKeyEncoder.java:48)
>
>                     at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.
> createInstance(AbstractRowKeyEncoder.java:48)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
>
>                     at org.apache.kylin.engine.spark.
> SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at org.apache.spark.api.java.JavaPairRDD$$anonfun$
> pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
>
>                     at scala.collection.Iterator$$
> anon$11.next(Iterator.scala:328)
>
>                     at org.apache.spark.util.collection.ExternalSorter.
> insertAll(ExternalSorter.scala:191)
>
>                     at org.apache.spark.shuffle.sort.
> SortShuffleWriter.write(SortShuffleWriter.scala:64)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:73)
>
>                     at org.apache.spark.scheduler.ShuffleMapTask.runTask(
> ShuffleMapTask.scala:41)
>
>                     at org.apache.spark.scheduler.Task.run(Task.scala:89)
>
>                     at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:227)
>
>                     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
>                     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>                     at java.lang.Thread.run(Thread.java:745)
>
>
>
> Any suggestions would help.
>
>
>
> Thanks,
>
> Zhuoran
>



-- 
Best regards,

Shaofeng Shi 史少锋