mismatch hdfs addr error when using kylin2.1 on two hadoop cluster

[朱真龙 <18...@qq.com>]

thank you for your attention first,I am a chinses kylin user that always read english documents but write little, and i know most of kylin developer are chinese 。so ，if you don't know what i mean，i will describe again in chinese。


     I am using kylin2.1.0 between two hadoop cluster （all configured HA） and hadoop have same version(2.7.1).  by now runing well with sample model  , but  not good in my own model which has many  columns encoding with dict。like this：



when  building cube , got error like this:



after looked into source code that the error metioned, find that the dictionary got hdfs path from hive table desc , and when checking before load these hfiles into hbase ,found hfiles and hbase table are on different hdfs cluster .


apache-kylin-2.1.0-src\apache-kylin-2.1.0\core-cube\src\main\java\org\apache\kylin\cube\CubeManager





so, could you tell me how could i do on this case ?

Re: mismatch hdfs addr error when using kylin2.1 on two hadoop cluster

[ShaoFeng Shi <sh...@apache.org>]

When configure Kylin with two cluster (one for hive/mapreduce, the other
for hbase), need:

1. in kylin node, the core-site.xml uses Hive cluster's HDFS as default FS.
2. in kylin.properties, configure "kylin.storage.hbase.cluster-fs" to the
HDFS of HBase cluster

In your case, I guess you didn't configure 2); please double check

2017-12-22 16:10 GMT+08:00 jxs <jx...@126.com>:

> I also deploy kylin on two small EMR clusters for resource isolation, I
> used MySQL for Hive metastore and shared the same hive configuration
> between the two clusters.
> You may try this.
>
>
>
> 在2017年12月22 13时08分, "朱真龙"<18...@qq.com>写道:
>
>
>     thank you for your attention first,I am a chinses kylin user that
> always read english documents but write little, and i know most of kylin
> developer are chinese 。so ，if you don't know what i mean，i will describe
> again in chinese。
>
>      I am using kylin2.1.0 between two hadoop cluster （all configured HA）
> and hadoop have same version(2.7.1).  by now runing well with sample model
> , but  not good in my own model which has many  columns encoding with
> dict。like this：
>
> when  building cube , got error like this:
>
> after looked into source code that the error metioned, find that the
> dictionary got hdfs path from hive table desc , and when checking before
> load these hfiles into hbase ,found hfiles and hbase table are on different
> hdfs cluster .
>
> apache-kylin-2.1.0-src\apache-kylin-2.1.0\core-cube\src\
> main\java\org\apache\kylin\cube\CubeManager
>
>
> so, could you tell me how could i do on this case ?
>
>
>
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Re:mismatch hdfs addr error when using kylin2.1 on two hadoop cluster

[jxs <jx...@126.com>]

I also deploy kylin on two small EMR clusters for resource isolation, I used MySQL for Hive metastore and shared the same hive configuration between the two clusters.
You may try this.







在2017年12月22 13时08分, "朱真龙"<18...@qq.com>写道:


    thank you for your attention first,I am a chinses kylin user that always read english documents but write little, and i know most of kylin developer are chinese 。so ，if you don't know what i mean，i will describe again in chinese。


     I am using kylin2.1.0 between two hadoop cluster （all configured HA） and hadoop have same version(2.7.1).  by now runing well with sample model  , but  not good in my own model which has many  columns encoding with dict。like this：


when  building cube , got error like this:


after looked into source code that the error metioned, find that the dictionary got hdfs path from hive table desc , and when checking before load these hfiles into hbase ,found hfiles and hbase table are on different hdfs cluster .


apache-kylin-2.1.0-src\apache-kylin-2.1.0\core-cube\src\main\java\org\apache\kylin\cube\CubeManager




so, could you tell me how could i do on this case ?