You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "dockj (JIRA)" <ji...@apache.org> on 2019/07/22 07:58:00 UTC
[jira] [Commented] (KYLIN-4105) use spark engine to build cube on
step Convert Cuboid Data to HFile failed
[ https://issues.apache.org/jira/browse/KYLIN-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889982#comment-16889982 ]
dockj commented on KYLIN-4105:
------------------------------
use spark engine build cube on step Convert Cuboid Data to HFile ，must invoke MeasureTypeFactory.init() to get custom MesureType。method call link:
+MeasureTypeFactory.init()->KylinConfig.getInstanceFromEnv()->KylinConfig.buildSiteProperties()->KylinConfig.buildSiteOrderedProps()->KylinConfig.getSitePropertiesFile()+
when yarn cluster set KYLIN_HOME or KYLIN_CONF environment variable，method KylinConfig.getSitePropertiesFile() will according to KYLIN_HOME or KYLIN_CONF to find kylin.properties，but yarn cluster host just set KYLIN_HOME or KYLIN_CONF,no kylin.properties file.method KylinConfig.buildSiteOrderedProps() will throws java.lang.RuntimeException: fail to locate kylin.properties。but method MeasureTypeFactory.init() just catch KylinConfigCannotInitException,finally cause use spark engine to build cube on step Convert Cuboid Data to HFile fail,maybe can expand method MeasureTypeFactory.init() KylinConfigCannotInitException to RuntimeException.
> use spark engine to build cube on step Convert Cuboid Data to HFile failed
> --------------------------------------------------------------------------
>
> Key: KYLIN-4105
> URL: https://issues.apache.org/jira/browse/KYLIN-4105
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v2.4.0, v2.4.1, v2.5.0, v2.6.0, v2.6.1, v2.6.2
> Reporter: dockj
> Priority: Minor
>
> use spark engine to build cube on step Convert Cuboid Data to HFile。visit yarn logs has this exception:
> 19/05/05 17:36:42 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, spslave4.bigdata.ly, executor 1): java.lang.ExceptionInInitializerError
> at org.apache.kylin.metadata.datatype.DataType.<clinit>(DataType.java:134)
> at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
> at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1630)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1714)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at java.util.HashSet.readObject(HashSet.java:333)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1909)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:85)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: fail to locate kylin.properties
> at org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:378)
> at org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:354)
> at org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:134)
> at org.apache.kylin.measure.MeasureTypeFactory.init(MeasureTypeFactory.java:120)
> at org.apache.kylin.measure.MeasureTypeFactory.<clinit>(MeasureTypeFactory.java:98)
> ... 71 more
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)