You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kylin.apache.org by "Wang Lei (北京研发中心.112456)" <al...@sohu-inc.com> on 2016/02/03 07:28:08 UTC

kylin.metadata.url with hbase namespace problem

Hi:
         We are using kylin 1.2 with hbase 0.98.16.1 and we have some problem with kylin.metadata.url.

         Env:
                   Kylin downloaded from https://dist.apache.org/repos/dist/release/kylin/apache-kylin-1.2/apache-kylin-1.2-bin.tar.gz
                   Hbase: 0.98.16.1 from community

         conf/kylin.properties
                  kylin.metadata.url=ALALEI:kylin_metadata@hbase

         when building sample cube，step 1 hangs on:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: Pathname /user/alalei/kylin/ALALEI:kylin_metadata/kylin-c39f5ba8-37cf-4da9-a995-b71fce988259/kylin_intermediate_kylin_sales_cube_desc_19700101000000_20160201000000_c39f5ba8_37cf_4da9_a995_b71fce988259 from /user/alalei/kylin/ALALEI:kylin_metadata/kylin-c39f5ba8-37cf-4da9-a995-b71fce988259/kylin_intermediate_kylin_sales_cube_desc_19700101000000_20160201000000_c39f5ba8_37cf_4da9_a995_b71fce988259 is not a valid DFS filename.)

        at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:90)
        at org.apache.kylin.job.common.ShellExecutable.doWork(ShellExecutable.java:52)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)


         any help will be appreciated !!


alaleiwang

Re: kylin.metadata.url with hbase namespace problem

Posted by hongbin ma <ma...@apache.org>.

BTW, It's welcomed if you can contribute a patch:)

On Wed, Feb 3, 2016 at 4:40 PM, hongbin ma <ma...@apache.org> wrote:

> is there a way to change default hbase namespace in configuration files
> like "hbase-site.xml"? then you don't have to do it in kylin.properties. If
> not I don't think there is a workaround, you might need to change code and
> compile kylin by yourself.
>
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: kylin.metadata.url with hbase namespace problem

Posted by hongbin ma <ma...@apache.org>.

is there a way to change default hbase namespace in configuration files
like "hbase-site.xml"? then you don't have to do it in kylin.properties. If
not I don't think there is a workaround, you might need to change code and
compile kylin by yourself.




-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: kylin.metadata.url with hbase namespace problem

Posted by alaleiwang <al...@sohu-inc.com>.

the above kylin.metadata.url with namespace works fine with hbase
operation,and i can see hbase new tables created under namespace "ALALEI"

the error happens in
./job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java with function
generateCreateTableStatement,where the second parameter storageDfsDir
defined in getMetadataUrlPrefix from
./common/src/main/java/org/apache/kylin/common/KylinConfig.java 

hdfs don't allow special colon in hdfs url path,so hive related create table
ddl with location can't be executed

any more help to support such usage case?
      


--
View this message in context: http://apache-kylin.74782.x6.nabble.com/kylin-metadata-url-with-hbase-namespace-problem-tp3577p3586.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: kylin.metadata.url with hbase namespace problem

Posted by hongbin ma <ma...@apache.org>.

one of the concerns is that we'll append the metadata_url to the hdfs
working dir path, and I guess hdfs path does not allow ":"

On Wed, Feb 3, 2016 at 3:16 PM, hongbin ma <ma...@apache.org> wrote:

> I'm not aware of such usage, where did you find any supportive documents
> on this?
> To support hbase namespace we might need a patch on this.
>
> On Wed, Feb 3, 2016 at 2:56 PM, alaleiwang <al...@sohu-inc.com>
> wrote:
>
>> because i want to change hbase namespace from "default" to "ALALEI",for
>> security reason
>>
>> --
>> View this message in context:
>> http://apache-kylin.74782.x6.nabble.com/kylin-metadata-url-with-hbase-namespace-problem-tp3577p3583.html
>> Sent from the Apache Kylin mailing list archive at Nabble.com.
>>
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: kylin.metadata.url with hbase namespace problem

Posted by hongbin ma <ma...@apache.org>.

I'm not aware of such usage, where did you find any supportive documents on
this?
To support hbase namespace we might need a patch on this.

On Wed, Feb 3, 2016 at 2:56 PM, alaleiwang <al...@sohu-inc.com> wrote:

> because i want to change hbase namespace from "default" to "ALALEI",for
> security reason
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/kylin-metadata-url-with-hbase-namespace-problem-tp3577p3583.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

Re: kylin.metadata.url with hbase namespace problem

Posted by alaleiwang <al...@sohu-inc.com>.

because i want to change hbase namespace from "default" to "ALALEI",for
security reason

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/kylin-metadata-url-with-hbase-namespace-problem-tp3577p3583.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: kylin.metadata.url with hbase namespace problem

Posted by hongbin ma <ma...@apache.org>.

*ALALEI:*kylin_metadata
@base


why do you need to append the prefix "ALALEI:" ? the char '':" may cause
troubles in unexpected places



-- 
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone