You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Bin.Li" <bi...@qq.com> on 2016/03/08 08:39:50 UTC
How can I setup kylin if hbase running on different cluster with hive and mapreduce
hello,
I'm trying to setup kylin(v1.2.0) on two different clusters,
but got error when I build a test cube:
Wrong FS: hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14, expected: hdfs://hbasefs
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:191) at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:347) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:426) at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:666) at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3621) at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3527) at org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3262) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29499) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012) ... 5 more# HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020 # leave empty if hbase running on same cluster with hive and mapreduce # kylin.hbase.cluster.fs= kylin.hbase.cluster.fs=hdfs://hmaster-addr:8020 # kylin.hbase.cluster.fs=hdfs://hbasefs:8020 # kylin.hbase.cluster.fs=hdfs://hbasefs/hbase # kylin.hbase.cluster.fs=hdfs://hbasefs
tks!
bin.li
Re: How can I setup kylin if hbase running on different cluster with hive and mapreduce
Posted by Yerui Sun <su...@gmail.com>.
Please try the following steps one by one:
1. keep kylin client and hmaster with same ‘hbase.rootdir’ config, like 'hdfs://hmaster-addr:8020' <hdfs://hmaster-addr:8020'>; <>
2. restart hmaster, to reload the config;
3. restart all regionservers, to get the config from hmaster;
4. restart kylin client;
5. discard the current cube building job, and build the cube again;
> ADD: if I put "kylin.hbase.cluster.fs=hdfs://hbasefs", it raise an "unknow host hbasefs” exception <>Seems that you’re using Namenode HA, right?
To enable this, you need have the name service related configs on kylin client, both in hdfs-site.xml and kylin_job_conf.xml.
However, if you’ve enable hadoop security, there’s maybe renew delegationToken issue in RM, please have a try and let me know whether it works.
> 在 2016年3月8日，17:19，Bin.Li <bi...@qq.com> 写道：
>
> hello yerui Sun,
>
> I've keep them same(hdfs://hmaster-addr:8020 <>) and restart kylin and rebuild test_cube
> <>
>
> <>
> <><property>
> <name>hbase.rootdir</name>
> <value>hdfs://hmaster-addr:8020</value>
> </property>
>
> but still got the same error:
> <> <> <> <>Wrong FS: <>hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14 <>, expected: hdfs://hbasefs <>
>
> ADD: if I put "kylin.hbase.cluster.fs=hdfs://hbasefs", it raise an "unknow host hbasefs" exception <>
> <>
> <>
>
> What's the problem??
>
> thks!!
>
> bin.li
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Yerui Sun";<su...@gmail.com>;
> 发送时间: 2016年3月8日(星期二) 下午4:59
> 收件人: "user"<us...@kylin.apache.org>;
> 主题: Re: How can I setup kylin if hbase running on different cluster with hive and mapreduce
>
> The reason of exception is that kylin using hdfs://hmaster-addr:8020 <>， but your hbase region server using hdfs://hbasefs <>，which determined by hmaster ‘hbase.rootdir’ config. Please keep both them same, and retry.
>
>> 在 2016年3月8日，15:39，Bin.Li <bingleessdut@qq.com <ma...@qq.com>> 写道：
>>
>> hello,
>>
>> I'm trying to setup kylin(v1.2.0) on two different clusters,
>> but got error when I build a test cube:
>>
>> Wrong FS: hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14 <>, expected: hdfs://hbasefs <>
>> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:191)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:102)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
>> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
>> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
>> at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:347)
>> at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:426)
>> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:666)
>> at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3621)
>> at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3527)
>> at org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3262)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29499)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
>> ... 5 more
>> # HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020 <>
>> # leave empty if hbase running on same cluster with hive and mapreduce
>> # kylin.hbase.cluster.fs=
>> kylin.hbase.cluster.fs=hdfs://hmaster-addr:8020 <>
>> # kylin.hbase.cluster.fs=hdfs://hbasefs:8020 <>
>> # kylin.hbase.cluster.fs=hdfs://hbasefs/hbase <>
>> # kylin.hbase.cluster.fs=hdfs://hbasefs <>
>>
>> tks!
>>
>> bin.li
>
回复： How can I setup kylin if hbase running on different cluster with hive and mapreduce
Posted by "Bin.Li" <bi...@qq.com>.
hello yerui Sun,
I've keep them same(hdfs://hmaster-addr:8020) and restart kylin and rebuild test_cube
<property>
<name>hbase.rootdir</name>
<value>hdfs://hmaster-addr:8020</value>
</property>
but still got the same error:
Wrong FS: hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14, expected: hdfs://hbasefs
ADD: if I put "kylin.hbase.cluster.fs=hdfs://hbasefs", it raise an "unknow host hbasefs" exception
What's the problem??
thks!!
bin.li
------------------ 原始邮件 ------------------
发件人: "Yerui Sun";<su...@gmail.com>;
发送时间: 2016年3月8日(星期二) 下午4:59
收件人: "user"<us...@kylin.apache.org>;
主题: Re: How can I setup kylin if hbase running on different cluster with hive and mapreduce
The reason of exception is that kylin using hdfs://hmaster-addr:8020， but your hbase region server using hdfs://hbasefs，which determined by hmaster ‘hbase.rootdir’ config. Please keep both them same, and retry.
在 2016年3月8日，15:39，Bin.Li <bi...@qq.com> 写道：
hello,
I'm trying to setup kylin(v1.2.0) on two different clusters,
but got error when I build a test cube:
Wrong FS: hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14, expected: hdfs://hbasefs
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:191) at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:102) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:347) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:426) at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:666) at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3621) at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3527) at org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3262) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29499) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012) ... 5 more# HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020 # leave empty if hbase running on same cluster with hive and mapreduce # kylin.hbase.cluster.fs= kylin.hbase.cluster.fs=hdfs://hmaster-addr:8020 # kylin.hbase.cluster.fs=hdfs://hbasefs:8020 # kylin.hbase.cluster.fs=hdfs://hbasefs/hbase # kylin.hbase.cluster.fs=hdfs://hbasefs
tks!
bin.li
Re: How can I setup kylin if hbase running on different cluster with hive and mapreduce
Posted by Yerui Sun <su...@gmail.com>.
The reason of exception is that kylin using hdfs://hmaster-addr:8020 <hdfs://hmaster-addr:8020>， but your hbase region server using hdfs://hbasefs <hdfs://hbasefs>，which determined by hmaster ‘hbase.rootdir’ config. Please keep both them same, and retry.
> 在 2016年3月8日，15:39，Bin.Li <bi...@qq.com> 写道：
>
> hello,
>
> I'm trying to setup kylin(v1.2.0) on two different clusters,
> but got error when I build a test cube:
>
> Wrong FS: hdfs://hmaster-addr:8020/data/kylin/kylin_metadata/kylin-bf5dace3-e885-4e47-9824-8fb9f778b590/test_cube/hfile/F1/ba97a8bdfda444ff8a00ac77ff6cda14, expected: hdfs://hbasefs
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:191)
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:102)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
> at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:347)
> at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:426)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:666)
> at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3621)
> at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3527)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFile(HRegionServer.java:3262)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29499)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> ... 5 more
> # HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020
> # leave empty if hbase running on same cluster with hive and mapreduce
> # kylin.hbase.cluster.fs=
> kylin.hbase.cluster.fs=hdfs://hmaster-addr:8020
> # kylin.hbase.cluster.fs=hdfs://hbasefs:8020
> # kylin.hbase.cluster.fs=hdfs://hbasefs/hbase
> # kylin.hbase.cluster.fs=hdfs://hbasefs
>
> tks!
>
> bin.li