You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shaofeng SHI (JIRA)" <ji...@apache.org> on 2018/06/01 07:08:00 UTC
[jira] [Commented] (KYLIN-3137) Spark cubing without hive-site.xml
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497656#comment-16497656 ]
Shaofeng SHI commented on KYLIN-3137:
-------------------------------------
I updated the code to directly parse the sequence files into RDD for cubing, then without hive context.
I have tested it can work. This will make the deploy much easier.
> Spark cubing without hive-site.xml
> ----------------------------------
>
> Key: KYLIN-3137
> URL: https://issues.apache.org/jira/browse/KYLIN-3137
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Others, Spark Engine
> Affects Versions: v2.2.0
> Reporter: Ruslan Dautkhanov
> Assignee: Shaofeng SHI
> Priority: Major
> Labels: cdh, cloudera, configuration, hive
> Fix For: v2.4.0
>
>
> Getting following exception while trying to build a cube
> {noformat}
> java.lang.RuntimeException: Cannot find hive-site.xml in kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you must set kylin.env.hadoop-conf-dir to a dir which contains at least core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
> at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I am using Kylin binaries for CDH downloaded from kylin.apache.org.
> Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's distribution for Hadoop.
> hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf
> The other four files:
> core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
> can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in /etc/hive/conf .
> Would be great to have this adjusted for CDH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)