You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Nikodimos Nikolaidis <ni...@csd.auth.gr> on 2019/02/11 14:44:09 UTC
Failed to read kylin_hive_conf.xml
Hello,
I am using latest (2.6) version of Kylin and I am following instructions
from https://kylin.apache.org/docs/tutorial/cube_streaming.html to build
a demo streaming cube.
At a BUILD CUBE job, I can see in monitor tab in "Hive Cleanup" step an
error occurs by the following java exception:
java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:87)
at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Then if I try to run a SQL query in Insight tab the following exception
is thrown:
java.lang.RuntimeException: fail to locate kylin.properties
while executing SQL: "select minute_start, count(*), sum(amount),
sum(qty) from streaming_sales_table group by minute_start order by
minute_start LIMIT 50000"
and generally, it seems that it can't find any Kylin configuration files
afterwards.
I have set properly JAVA_HOME, HADOOP_HOME, HIVE_HOME, HBASE_HOME,
KYLIN_HOME environment variables.
What could be the reason for that?
Re: Failed to read kylin_hive_conf.xml
Posted by Nikodimos Nikolaidis <ni...@csd.auth.gr>.
I can say for sure that KYLIN_CONF property is set to "meta" during a
build job (using MR as engine), because when I first start kylin server
and before any build jobs, I can run a query in Insight tab, while after
a (failed) build job, there will be an error locating "kylin.properties"
for queries and about everything else.
On 14/02/2019 00:38, ShaoFeng Shi wrote:
> Hi Nikodimos,
>
> Interesting finding; The method of
> "AbstractHadoopJob#loadKylinPropsAndMetadata" only be invoked from a
> Hadoop container (mapper/reducer), so the "KYILN_CONF" only be set out
> of Kylin's main process; Can you double check your local environment,
> see whether "KYLIN_CONF" has already been set, like some shell script,
> or bash_profile, etc?
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Work email: shaofeng.shi@kyligence.io
> <ma...@kyligence.io>
> Kyligence Inc: https://kyligence.io/
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> <ma...@kylin.apache.org>
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> <ma...@kylin.apache.org>
>
>
>
>
> Nikodimos Nikolaidis <nikniknik@csd.auth.gr
> <ma...@csd.auth.gr>> 于2019年2月13日周三 上午7:36写道：
>
> After compiling master branch with your latest commit, which
> prints `confFile.getAbsolutePath()`, the ouput was "Failed to read
> kylin_hive_conf.xml at
> '/opt/apache-kylin-2.6.0-bin/meta/kylin_hive_conf.xml'".
>
> It seems that KYLIN_CONF property is set to $KYLIN_HOME/meta
> (possibly at
> https://github.com/apache/kylin/blob/master/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java#L478
> ).
>
> By creating a link `meta` to `conf` in $KYLIN_HOME, build job
> worked without errors. But, why is KYLIN_CONF property set like
> that? Is there an error in my configuration or is it a bug?
>
> thanks by the way
>
> On 2/11/19 8:55 PM, ShaoFeng Shi wrote:
>> Usually, this is an environment issue I think; The exception
>> didn't clearly print the real path, could you modify this line
>> code to let it print the "confFile"?
>>
>> https://github.com/apache/kylin/blob/master/core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java#L87
>>
>> Besides, please also check the kylin.log before and after the
>> error, there should be some debugging logs.
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Work email: shaofeng.shi@kyligence.io
>> <ma...@kyligence.io>
>> Kyligence Inc: https://kyligence.io/
>>
>> Apache Kylin FAQ:
>> https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> <ma...@kylin.apache.org>
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>> <ma...@kylin.apache.org>
>>
>>
>>
>>
>> Nikodimos Nikolaidis <nikniknik@csd.auth.gr
>> <ma...@csd.auth.gr>> 于2019年2月11日周一 下午10:44写道：
>>
>> Hello,
>>
>> I am using latest (2.6) version of Kylin and I am following
>> instructions from
>> https://kylin.apache.org/docs/tutorial/cube_streaming.html to
>> build a demo streaming cube.
>>
>> At a BUILD CUBE job, I can see in monitor tab in "Hive
>> Cleanup" step an error occurs by the following java exception:
>>
>> java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
>> at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:87)
>> at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
>> at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
>> at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
>> at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Then if I try to run a SQL query in Insight tab the following
>> exception is thrown:
>>
>> java.lang.RuntimeException: fail to locate kylin.properties
>> while executing SQL: "select minute_start, count(*), sum(amount),
>> sum(qty) from streaming_sales_table group by minute_start order by
>> minute_start LIMIT 50000"
>>
>> and generally, it seems that it can't find any Kylin
>> configuration files afterwards.
>>
>> I have set properly JAVA_HOME, HADOOP_HOME, HIVE_HOME,
>> HBASE_HOME, KYLIN_HOME environment variables.
>>
>> What could be the reason for that?
>>
Re: Failed to read kylin_hive_conf.xml
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Nikodimos,
Interesting finding; The method of
"AbstractHadoopJob#loadKylinPropsAndMetadata" only be invoked from a Hadoop
container (mapper/reducer), so the "KYILN_CONF" only be set out of Kylin's
main process; Can you double check your local environment, see whether
"KYLIN_CONF" has already been set, like some shell script, or bash_profile,
etc?
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Work email: shaofeng.shi@kyligence.io
Kyligence Inc: https://kyligence.io/
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Nikodimos Nikolaidis <ni...@csd.auth.gr> 于2019年2月13日周三 上午7:36写道：
> After compiling master branch with your latest commit, which prints `
> confFile.getAbsolutePath()`, the ouput was "Failed to read
> kylin_hive_conf.xml at
> '/opt/apache-kylin-2.6.0-bin/meta/kylin_hive_conf.xml'".
>
> It seems that KYLIN_CONF property is set to $KYLIN_HOME/meta (possibly at
> https://github.com/apache/kylin/blob/master/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java#L478
> ).
>
> By creating a link `meta` to `conf` in $KYLIN_HOME, build job worked
> without errors. But, why is KYLIN_CONF property set like that? Is there an
> error in my configuration or is it a bug?
>
> thanks by the way
> On 2/11/19 8:55 PM, ShaoFeng Shi wrote:
>
> Usually, this is an environment issue I think; The exception didn't
> clearly print the real path, could you modify this line code to let it
> print the "confFile"?
>
>
> https://github.com/apache/kylin/blob/master/core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java#L87
>
> Besides, please also check the kylin.log before and after the error, there
> should be some debugging logs.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Work email: shaofeng.shi@kyligence.io
> Kyligence Inc: https://kyligence.io/
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Nikodimos Nikolaidis <ni...@csd.auth.gr> 于2019年2月11日周一 下午10:44写道：
>
>> Hello,
>>
>> I am using latest (2.6) version of Kylin and I am following instructions
>> from https://kylin.apache.org/docs/tutorial/cube_streaming.html to build
>> a demo streaming cube.
>>
>> At a BUILD CUBE job, I can see in monitor tab in "Hive Cleanup" step an
>> error occurs by the following java exception:
>>
>> java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
>> at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:87)
>> at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
>> at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
>> at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
>> at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> Then if I try to run a SQL query in Insight tab the following exception
>> is thrown:
>>
>> java.lang.RuntimeException: fail to locate kylin.properties
>> while executing SQL: "select minute_start, count(*), sum(amount),
>> sum(qty) from streaming_sales_table group by minute_start order by
>> minute_start LIMIT 50000"
>>
>>
>> and generally, it seems that it can't find any Kylin configuration files
>> afterwards.
>>
>> I have set properly JAVA_HOME, HADOOP_HOME, HIVE_HOME, HBASE_HOME,
>> KYLIN_HOME environment variables.
>>
>> What could be the reason for that?
>>
>>
>
Re: Failed to read kylin_hive_conf.xml
Posted by Nikodimos Nikolaidis <ni...@csd.auth.gr>.
After compiling master branch with your latest commit, which prints
`confFile.getAbsolutePath()`, the ouput was "Failed to read
kylin_hive_conf.xml at
'/opt/apache-kylin-2.6.0-bin/meta/kylin_hive_conf.xml'".
It seems that KYLIN_CONF property is set to $KYLIN_HOME/meta (possibly
at
https://github.com/apache/kylin/blob/master/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java#L478
).
By creating a link `meta` to `conf` in $KYLIN_HOME, build job worked
without errors. But, why is KYLIN_CONF property set like that? Is there
an error in my configuration or is it a bug?
thanks by the way
On 2/11/19 8:55 PM, ShaoFeng Shi wrote:
> Usually, this is an environment issue I think; The exception didn't
> clearly print the real path, could you modify this line code to let it
> print the "confFile"?
>
> https://github.com/apache/kylin/blob/master/core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java#L87
>
> Besides, please also check the kylin.log before and after the error,
> there should be some debugging logs.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Work email: shaofeng.shi@kyligence.io
> <ma...@kyligence.io>
> Kyligence Inc: https://kyligence.io/
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> <ma...@kylin.apache.org>
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> <ma...@kylin.apache.org>
>
>
>
>
> Nikodimos Nikolaidis <nikniknik@csd.auth.gr
> <ma...@csd.auth.gr>> 于2019年2月11日周一 下午10:44写道：
>
> Hello,
>
> I am using latest (2.6) version of Kylin and I am following
> instructions from
> https://kylin.apache.org/docs/tutorial/cube_streaming.html to
> build a demo streaming cube.
>
> At a BUILD CUBE job, I can see in monitor tab in "Hive Cleanup"
> step an error occurs by the following java exception:
>
> java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
> at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:87)
> at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
> at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
> at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
> at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Then if I try to run a SQL query in Insight tab the following
> exception is thrown:
>
> java.lang.RuntimeException: fail to locate kylin.properties
> while executing SQL: "select minute_start, count(*), sum(amount),
> sum(qty) from streaming_sales_table group by minute_start order by
> minute_start LIMIT 50000"
>
> and generally, it seems that it can't find any Kylin configuration
> files afterwards.
>
> I have set properly JAVA_HOME, HADOOP_HOME, HIVE_HOME, HBASE_HOME,
> KYLIN_HOME environment variables.
>
> What could be the reason for that?
>
>
>
Re: Failed to read kylin_hive_conf.xml
Posted by ShaoFeng Shi <sh...@apache.org>.
Usually, this is an environment issue I think; The exception didn't clearly
print the real path, could you modify this line code to let it print the "
confFile"?
https://github.com/apache/kylin/blob/master/core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java#L87
Besides, please also check the kylin.log before and after the error, there
should be some debugging logs.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Work email: shaofeng.shi@kyligence.io
Kyligence Inc: https://kyligence.io/
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Nikodimos Nikolaidis <ni...@csd.auth.gr> 于2019年2月11日周一 下午10:44写道：
> Hello,
>
> I am using latest (2.6) version of Kylin and I am following instructions
> from https://kylin.apache.org/docs/tutorial/cube_streaming.html to build
> a demo streaming cube.
>
> At a BUILD CUBE job, I can see in monitor tab in "Hive Cleanup" step an
> error occurs by the following java exception:
>
> java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
> at org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:87)
> at org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:61)
> at org.apache.kylin.common.util.HiveCmdBuilder.<init>(HiveCmdBuilder.java:48)
> at org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
> at org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Then if I try to run a SQL query in Insight tab the following exception is
> thrown:
>
> java.lang.RuntimeException: fail to locate kylin.properties
> while executing SQL: "select minute_start, count(*), sum(amount),
> sum(qty) from streaming_sales_table group by minute_start order by
> minute_start LIMIT 50000"
>
>
> and generally, it seems that it can't find any Kylin configuration files
> afterwards.
>
> I have set properly JAVA_HOME, HADOOP_HOME, HIVE_HOME, HBASE_HOME,
> KYLIN_HOME environment variables.
>
> What could be the reason for that?
>
>