One question about set up apache Kylin at Azure Hdinsight cluster

[Jiang Xu <ji...@citrix.com>]

Hi guys,



I am trying to set up the apache Kylin at Azure Hdinsight cluster, and I followed the procedure on below link from Apache Kylin official web, and I have some questions, and need your help to give some suggestions and comments.

Link I checked: http://kylin.apache.org/docs/tutorial/cube_spark.html, the example is for Hortonworks HDP 2.4 Sandbox VM, and what I am using is Azure Hdinsight cluster with “spark” type.

Questions:

1, I set up one Azure Hdinsight cluster with “spark” type, is it ok to install apache Kylin?

2, for the configuration, I have one question for your guy, I copied the below screen for your reference, for the first line in the below rectangle, for “Kylin,engine.spark-conf.spark.yarn.archive=hdfs://sandox.hortworks.com:8020/kylin/spark/spark-libs.jar”, how to configure the namenode for azure hdinsight cluster case?

[cid:C6977F40-12C6-4C52-BAF1-D9F4E5149916]


BTW:

I attached the error log when start the Kylin:


####
root@hn0-kylina:/usr/local/apache-kylin-2.4.0-bin-hbase1x# $KYLIN_HOME/bin/kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/apache-kylin-2.4.0-bin-hbase1x
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
find: тАШ/usr/hdp/current/spark-client/jarsтАЩ: No such file or directory
spark jars not found
####

Regards,
Jiangxu

Re: One question about set up apache Kylin at Azure Hdinsight cluster

[ShaoFeng Shi <sh...@apache.org>]

Hello Jiang,

What's your HDInsight version? We verified HDI 3.5 and 3.6 are compatible,
not tested on other versions.

HDI already has "SPARK_HOME" system environment variable set, and pointed
to "/usr/hdp/current/spark-client". In this case, Kylin would use the
"SPARK_HOME" set in the system, not the embedded one. If the system Spark
is not a compatible version (v2.1), it may reports error. So suggest you to
explicitly set SPARK_HOME to KYLIN_HOME/spark before start Kylin.

Besides, when using community Spark on HDI, if it reports ClassNotFound,
you need copy some vendor specific jars from system spark.

For your second question, you can check the core-site.xml on HDI node, then
use the fully qualified name.


2018-08-03 10:10 GMT+08:00 Jiang Xu <ji...@citrix.com>:

> Hi guys,
>
>
>
> I am trying to set up the apache Kylin at Azure Hdinsight cluster, and I
> followed the procedure on below link from Apache Kylin official web, and I
> have some questions, and need your help to give some suggestions and
> comments.
>
> Link I checked: http://kylin.apache.org/docs/tutorial/cube_spark.html,
> the example is for Hortonworks HDP 2.4 Sandbox VM, and what I am using is
> Azure Hdinsight cluster with “spark” type.
>
> Questions:
>
> 1, I set up one Azure Hdinsight cluster with “spark” type, is it ok to
> install apache Kylin?
>
> 2, for the configuration, I have one question for your guy, I copied the
> below screen for your reference, for the first line in the below rectangle,
> for “Kylin,engine.spark-conf.spark.yarn.archive=hdfs://
> sandox.hortworks.com:8020/kylin/spark/spark-libs.jar”, how to configure
> the namenode for azure hdinsight cluster case?
>
>
>
> BTW:
>
> I attached the error log when start the Kylin:
>
>
> ####
> root@hn0-kylina:/usr/local/apache-kylin-2.4.0-bin-hbase1x#
> $KYLIN_HOME/bin/kylin.sh start
> Retrieving hadoop conf dir...
> KYLIN_HOME is set to /usr/local/apache-kylin-2.4.0-bin-hbase1x
> Retrieving hive dependency...
> Retrieving hbase dependency...
> Retrieving hadoop conf dir...
> Retrieving kafka dependency...
> Retrieving Spark dependency...
> find: тАШ/usr/hdp/current/spark-client/jarsтАЩ: No such file or directory
> spark jars not found
> ####
>
> Regards,
> Jiangxu
>



-- 
Best regards,

Shaofeng Shi 史少锋