[jira] [Created] (KYLIN-4493) On HDP3 using spark to build dimension dictionary, it throws NoSuchMethodError.

["Zhichao Zhang (Jira)" <ji...@apache.org>]

Zhichao  Zhang created KYLIN-4493:
-------------------------------------

             Summary: On HDP3 using spark to build dimension dictionary, it throws NoSuchMethodError.
                 Key: KYLIN-4493
                 URL: https://issues.apache.org/jira/browse/KYLIN-4493
             Project: Kylin
          Issue Type: Bug
          Components: Spark Engine
    Affects Versions: v3.0.1, v3.0.2
            Reporter: Zhichao  Zhang


Test env:
{code:java}
 HDP 3.0.1.0-187
 Hadoop 3.1.1
 Hive 3.1.0
 Kylin 3.0.2
 Spark 2.3.2{code}
 

Problems:
 when use spark engine and turn on *'kylin.engine.spark-dimension-dictionary'*, it will throw error:
{code:java}
java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/conf/Configuration;)V
    at org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:164)
    at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78)
    at org.apache.kylin.source.hive.HiveTable.<init>(HiveTable.java:48)
    at org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:68)
    at org.apache.kylin.source.SourceManager.createReadableTable(SourceManager.java:145)
    at org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.buildSnapshotTable(SparkBuildDictionary.java:386)
    at org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.call(SparkBuildDictionary.java:367)
    at org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.call(SparkBuildDictionary.java:325)
    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
    at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:462)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
    at scala.collection.AbstractIterator.to(Iterator.scala:1336)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:945)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:945)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)