You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 柯南 <zz...@qq.com> on 2017/09/28 15:57:39 UTC
回复： A problem about Global Dictionary and cube merge
Thank you for your reply!
I know this command.But Because its segment could not be merged, even if run this command ,the hbase tables could not be cleaned.
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年9月28日(星期四) 晚上9:29
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: A problem about Global Dictionary and cube merge
You need run this:
https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
> hi,all：
> Because of our yarn and hbase`s version limited,we deploy apache
> kylin 1.6.0. For the correct result of 'count(distinct A_col)' across
> segments, we must use Global Dictionary to A_col. When the cube contain
> configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder', its
> segment could not be merged.(AppendTrieDictionary class don`t support
> getValueBytesFromIdImpl method and error is below) We will build the cube
> every day, that means a new hbase table will be created every day. And the
> segment could not be merged, that means the hbase tables could not be
> cleaned. That`t terrible.Give me some advice?
> Thank you! Looking forward to your reply.
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: Re: A problem about Global Dictionary and cube merge
Posted by ShaoFeng Shi <sh...@apache.org>.
For int column, if you use "integer" encoding for it, there is no such
issue. Besides, you also don't need specify GlobalDicionary if the column
value is already interger.
2017-09-29 13:52 GMT+08:00 崔苗 <cu...@danale.com>:
> how about int column? Can I use a integer column in on cube both as
> dimension and distinct count?
> 在 2017-09-29 11:41:02，ShaoFeng Shi <sh...@apache.org> 写道：
> >Split that into two cubes: in one it acts as a dimension, in the other, it
> >acts as count distinct measure.
> >
> >2017-09-29 11:12 GMT+08:00 崔苗 <cu...@danale.com>:
> >
> >> if a column was used both as dimension and distinct count,what should
> be
> >> use to encode ?
> >> 在 2017-09-29 11:05:28，ShaoFeng Shi <sh...@apache.org> 写道：
> >> >Use GD for dimension encoding will cause query inaccurate, that should
> be
> >> >avoided.
> >> >
> >> >Please use dict or fixed_length encoding for that column.
> >> >
> >> >2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
> >> >
> >> >> Yes，I am careful to use GD. Could I have another way to instead of
> GD？
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ------------------ 原始邮件 ------------------
> >> >> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> >> >> 发送时间: 2017年9月29日(星期五) 上午9:41
> >> >> 收件人: "dev"<de...@kylin.apache.org>;
> >> >>
> >> >> 主题: Re: A problem about Global Dictionary and cube merge
> >> >>
> >> >>
> >> >>
> >> >> I see; Did you use GlobalDictionaryBuilder for a dimension column?
> The
> >> GD
> >> >> should only be used for count distinct measure, not suitable for
> >> dimension
> >> >> encoding.
> >> >>
> >> >> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
> >> >>
> >> >> > Thank you for your reply!
> >> >> > I know this command.But Because its segment could not be merged,
> >> even if
> >> >> > run this command ,the hbase tables could not be cleaned.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > ------------------ 原始邮件 ------------------
> >> >> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> >> >> > 发送时间: 2017年9月28日(星期四) 晚上9:29
> >> >> > 收件人: "dev"<de...@kylin.apache.org>;
> >> >> >
> >> >> > 主题: Re: A problem about Global Dictionary and cube merge
> >> >> >
> >> >> >
> >> >> >
> >> >> > You need run this:
> >> >> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
> >> >> >
> >> >> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
> >> >> >
> >> >> > > hi,all：
> >> >> > > Because of our yarn and hbase`s version limited,we deploy
> >> apache
> >> >> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)'
> >> across
> >> >> > > segments, we must use Global Dictionary to A_col. When the cube
> >> contain
> >> >> > > configuration of 'org.apache.kylin.dict.GlobalD
> ictionaryBuilder',
> >> its
> >> >> > > segment could not be merged.(AppendTrieDictionary class don`t
> >> support
> >> >> > > getValueBytesFromIdImpl method and error is below) We will build
> the
> >> >> cube
> >> >> > > every day, that means a new hbase table will be created every
> day.
> >> And
> >> >> > the
> >> >> > > segment could not be merged, that means the hbase tables could
> not
> >> be
> >> >> > > cleaned. That`t terrible.Give me some advice?
> >> >> > > Thank you! Looking forward to your reply.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> >
> >> >> > Shaofeng Shi 史少锋
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best regards,
> >> >>
> >> >> Shaofeng Shi 史少锋
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >Best regards,
> >> >
> >> >Shaofeng Shi 史少锋
> >>
> >>
> >>
> >
> >
> >--
> >Best regards,
> >
> >Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: A problem about Global Dictionary and cube merge
Posted by ShaoFeng Shi <sh...@apache.org>.
The issue is discussed in https://issues.apache.org/jira/browse/KYLIN-2679
2017-09-29 11:41 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Split that into two cubes: in one it acts as a dimension, in the other, it
> acts as count distinct measure.
>
> 2017-09-29 11:12 GMT+08:00 崔苗 <cu...@danale.com>:
>
>> if a column was used both as dimension and distinct count,what should be
>> use to encode ?
>> 在 2017-09-29 11:05:28，ShaoFeng Shi <sh...@apache.org> 写道：
>> >Use GD for dimension encoding will cause query inaccurate, that should be
>> >avoided.
>> >
>> >Please use dict or fixed_length encoding for that column.
>> >
>> >2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
>> >
>> >> Yes，I am careful to use GD. Could I have another way to instead of GD？
>> >>
>> >>
>> >>
>> >>
>> >> ------------------ 原始邮件 ------------------
>> >> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> >> 发送时间: 2017年9月29日(星期五) 上午9:41
>> >> 收件人: "dev"<de...@kylin.apache.org>;
>> >>
>> >> 主题: Re: A problem about Global Dictionary and cube merge
>> >>
>> >>
>> >>
>> >> I see; Did you use GlobalDictionaryBuilder for a dimension column? The
>> GD
>> >> should only be used for count distinct measure, not suitable for
>> dimension
>> >> encoding.
>> >>
>> >> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
>> >>
>> >> > Thank you for your reply!
>> >> > I know this command.But Because its segment could not be merged,
>> even if
>> >> > run this command ,the hbase tables could not be cleaned.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------ 原始邮件 ------------------
>> >> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> >> > 发送时间: 2017年9月28日(星期四) 晚上9:29
>> >> > 收件人: "dev"<de...@kylin.apache.org>;
>> >> >
>> >> > 主题: Re: A problem about Global Dictionary and cube merge
>> >> >
>> >> >
>> >> >
>> >> > You need run this:
>> >> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
>> >> >
>> >> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
>> >> >
>> >> > > hi,all：
>> >> > > Because of our yarn and hbase`s version limited,we deploy
>> apache
>> >> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)'
>> across
>> >> > > segments, we must use Global Dictionary to A_col. When the cube
>> contain
>> >> > > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder',
>> its
>> >> > > segment could not be merged.(AppendTrieDictionary class don`t
>> support
>> >> > > getValueBytesFromIdImpl method and error is below) We will build
>> the
>> >> cube
>> >> > > every day, that means a new hbase table will be created every day.
>> And
>> >> > the
>> >> > > segment could not be merged, that means the hbase tables could not
>> be
>> >> > > cleaned. That`t terrible.Give me some advice?
>> >> > > Thank you! Looking forward to your reply.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> >
>> >> > Shaofeng Shi 史少锋
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >>
>> >> Shaofeng Shi 史少锋
>> >>
>> >
>> >
>> >
>> >--
>> >Best regards,
>> >
>> >Shaofeng Shi 史少锋
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re:Re: Re: A problem about Global Dictionary and cube merge
Posted by 崔苗 <cu...@danale.com>.
how about int column? Can I use a integer column in on cube both as dimension and distinct count?
在 2017-09-29 11:41:02，ShaoFeng Shi <sh...@apache.org> 写道：
>Split that into two cubes: in one it acts as a dimension, in the other, it
>acts as count distinct measure.
>
>2017-09-29 11:12 GMT+08:00 崔苗 <cu...@danale.com>:
>
>> if a column was used both as dimension and distinct count,what should be
>> use to encode ?
>> 在 2017-09-29 11:05:28，ShaoFeng Shi <sh...@apache.org> 写道：
>> >Use GD for dimension encoding will cause query inaccurate, that should be
>> >avoided.
>> >
>> >Please use dict or fixed_length encoding for that column.
>> >
>> >2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
>> >
>> >> Yes，I am careful to use GD. Could I have another way to instead of GD？
>> >>
>> >>
>> >>
>> >>
>> >> ------------------ 原始邮件 ------------------
>> >> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> >> 发送时间: 2017年9月29日(星期五) 上午9:41
>> >> 收件人: "dev"<de...@kylin.apache.org>;
>> >>
>> >> 主题: Re: A problem about Global Dictionary and cube merge
>> >>
>> >>
>> >>
>> >> I see; Did you use GlobalDictionaryBuilder for a dimension column? The
>> GD
>> >> should only be used for count distinct measure, not suitable for
>> dimension
>> >> encoding.
>> >>
>> >> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
>> >>
>> >> > Thank you for your reply!
>> >> > I know this command.But Because its segment could not be merged,
>> even if
>> >> > run this command ,the hbase tables could not be cleaned.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ------------------ 原始邮件 ------------------
>> >> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> >> > 发送时间: 2017年9月28日(星期四) 晚上9:29
>> >> > 收件人: "dev"<de...@kylin.apache.org>;
>> >> >
>> >> > 主题: Re: A problem about Global Dictionary and cube merge
>> >> >
>> >> >
>> >> >
>> >> > You need run this:
>> >> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
>> >> >
>> >> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
>> >> >
>> >> > > hi,all：
>> >> > > Because of our yarn and hbase`s version limited,we deploy
>> apache
>> >> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)'
>> across
>> >> > > segments, we must use Global Dictionary to A_col. When the cube
>> contain
>> >> > > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder',
>> its
>> >> > > segment could not be merged.(AppendTrieDictionary class don`t
>> support
>> >> > > getValueBytesFromIdImpl method and error is below) We will build the
>> >> cube
>> >> > > every day, that means a new hbase table will be created every day.
>> And
>> >> > the
>> >> > > segment could not be merged, that means the hbase tables could not
>> be
>> >> > > cleaned. That`t terrible.Give me some advice?
>> >> > > Thank you! Looking forward to your reply.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> >
>> >> > Shaofeng Shi 史少锋
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >>
>> >> Shaofeng Shi 史少锋
>> >>
>> >
>> >
>> >
>> >--
>> >Best regards,
>> >
>> >Shaofeng Shi 史少锋
>>
>>
>>
>
>
>--
>Best regards,
>
>Shaofeng Shi 史少锋
Re: Re: A problem about Global Dictionary and cube merge
Posted by ShaoFeng Shi <sh...@apache.org>.
Split that into two cubes: in one it acts as a dimension, in the other, it
acts as count distinct measure.
2017-09-29 11:12 GMT+08:00 崔苗 <cu...@danale.com>:
> if a column was used both as dimension and distinct count,what should be
> use to encode ?
> 在 2017-09-29 11:05:28，ShaoFeng Shi <sh...@apache.org> 写道：
> >Use GD for dimension encoding will cause query inaccurate, that should be
> >avoided.
> >
> >Please use dict or fixed_length encoding for that column.
> >
> >2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
> >
> >> Yes，I am careful to use GD. Could I have another way to instead of GD？
> >>
> >>
> >>
> >>
> >> ------------------ 原始邮件 ------------------
> >> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> >> 发送时间: 2017年9月29日(星期五) 上午9:41
> >> 收件人: "dev"<de...@kylin.apache.org>;
> >>
> >> 主题: Re: A problem about Global Dictionary and cube merge
> >>
> >>
> >>
> >> I see; Did you use GlobalDictionaryBuilder for a dimension column? The
> GD
> >> should only be used for count distinct measure, not suitable for
> dimension
> >> encoding.
> >>
> >> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
> >>
> >> > Thank you for your reply!
> >> > I know this command.But Because its segment could not be merged,
> even if
> >> > run this command ,the hbase tables could not be cleaned.
> >> >
> >> >
> >> >
> >> >
> >> > ------------------ 原始邮件 ------------------
> >> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> >> > 发送时间: 2017年9月28日(星期四) 晚上9:29
> >> > 收件人: "dev"<de...@kylin.apache.org>;
> >> >
> >> > 主题: Re: A problem about Global Dictionary and cube merge
> >> >
> >> >
> >> >
> >> > You need run this:
> >> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
> >> >
> >> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
> >> >
> >> > > hi,all：
> >> > > Because of our yarn and hbase`s version limited,we deploy
> apache
> >> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)'
> across
> >> > > segments, we must use Global Dictionary to A_col. When the cube
> contain
> >> > > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder',
> its
> >> > > segment could not be merged.(AppendTrieDictionary class don`t
> support
> >> > > getValueBytesFromIdImpl method and error is below) We will build the
> >> cube
> >> > > every day, that means a new hbase table will be created every day.
> And
> >> > the
> >> > > segment could not be merged, that means the hbase tables could not
> be
> >> > > cleaned. That`t terrible.Give me some advice?
> >> > > Thank you! Looking forward to your reply.
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > Shaofeng Shi 史少锋
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >>
> >
> >
> >
> >--
> >Best regards,
> >
> >Shaofeng Shi 史少锋
>
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re:Re: A problem about Global Dictionary and cube merge
Posted by 崔苗 <cu...@danale.com>.
if a column was used both as dimension and distinct count,what should be use to encode ?
在 2017-09-29 11:05:28，ShaoFeng Shi <sh...@apache.org> 写道：
>Use GD for dimension encoding will cause query inaccurate, that should be
>avoided.
>
>Please use dict or fixed_length encoding for that column.
>
>2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
>
>> Yes，I am careful to use GD. Could I have another way to instead of GD？
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> 发送时间: 2017年9月29日(星期五) 上午9:41
>> 收件人: "dev"<de...@kylin.apache.org>;
>>
>> 主题: Re: A problem about Global Dictionary and cube merge
>>
>>
>>
>> I see; Did you use GlobalDictionaryBuilder for a dimension column? The GD
>> should only be used for count distinct measure, not suitable for dimension
>> encoding.
>>
>> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
>>
>> > Thank you for your reply!
>> > I know this command.But Because its segment could not be merged, even if
>> > run this command ,the hbase tables could not be cleaned.
>> >
>> >
>> >
>> >
>> > ------------------ 原始邮件 ------------------
>> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
>> > 发送时间: 2017年9月28日(星期四) 晚上9:29
>> > 收件人: "dev"<de...@kylin.apache.org>;
>> >
>> > 主题: Re: A problem about Global Dictionary and cube merge
>> >
>> >
>> >
>> > You need run this:
>> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
>> >
>> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
>> >
>> > > hi,all：
>> > > Because of our yarn and hbase`s version limited,we deploy apache
>> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)' across
>> > > segments, we must use Global Dictionary to A_col. When the cube contain
>> > > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder', its
>> > > segment could not be merged.(AppendTrieDictionary class don`t support
>> > > getValueBytesFromIdImpl method and error is below) We will build the
>> cube
>> > > every day, that means a new hbase table will be created every day. And
>> > the
>> > > segment could not be merged, that means the hbase tables could not be
>> > > cleaned. That`t terrible.Give me some advice?
>> > > Thank you! Looking forward to your reply.
>> >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > Shaofeng Shi 史少锋
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
>
>
>--
>Best regards,
>
>Shaofeng Shi 史少锋
Re: A problem about Global Dictionary and cube merge
Posted by ShaoFeng Shi <sh...@apache.org>.
Use GD for dimension encoding will cause query inaccurate, that should be
avoided.
Please use dict or fixed_length encoding for that column.
2017-09-29 10:37 GMT+08:00 柯南 <zz...@qq.com>:
> Yes，I am careful to use GD. Could I have another way to instead of GD？
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年9月29日(星期五) 上午9:41
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: A problem about Global Dictionary and cube merge
>
>
>
> I see; Did you use GlobalDictionaryBuilder for a dimension column? The GD
> should only be used for count distinct measure, not suitable for dimension
> encoding.
>
> 2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
>
> > Thank you for your reply!
> > I know this command.But Because its segment could not be merged, even if
> > run this command ,the hbase tables could not be cleaned.
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年9月28日(星期四) 晚上9:29
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: A problem about Global Dictionary and cube merge
> >
> >
> >
> > You need run this:
> > https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
> >
> > 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
> >
> > > hi,all：
> > > Because of our yarn and hbase`s version limited,we deploy apache
> > > kylin 1.6.0. For the correct result of 'count(distinct A_col)' across
> > > segments, we must use Global Dictionary to A_col. When the cube contain
> > > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder', its
> > > segment could not be merged.(AppendTrieDictionary class don`t support
> > > getValueBytesFromIdImpl method and error is below) We will build the
> cube
> > > every day, that means a new hbase table will be created every day. And
> > the
> > > segment could not be merged, that means the hbase tables could not be
> > > cleaned. That`t terrible.Give me some advice?
> > > Thank you! Looking forward to your reply.
> >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： A problem about Global Dictionary and cube merge
Posted by 柯南 <zz...@qq.com>.
Yes，I am careful to use GD. Could I have another way to instead of GD？
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年9月29日(星期五) 上午9:41
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: A problem about Global Dictionary and cube merge
I see; Did you use GlobalDictionaryBuilder for a dimension column? The GD
should only be used for count distinct measure, not suitable for dimension
encoding.
2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
> Thank you for your reply!
> I know this command.But Because its segment could not be merged, even if
> run this command ,the hbase tables could not be cleaned.
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年9月28日(星期四) 晚上9:29
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: A problem about Global Dictionary and cube merge
>
>
>
> You need run this:
> https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
>
> 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
>
> > hi,all：
> > Because of our yarn and hbase`s version limited,we deploy apache
> > kylin 1.6.0. For the correct result of 'count(distinct A_col)' across
> > segments, we must use Global Dictionary to A_col. When the cube contain
> > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder', its
> > segment could not be merged.(AppendTrieDictionary class don`t support
> > getValueBytesFromIdImpl method and error is below) We will build the cube
> > every day, that means a new hbase table will be created every day. And
> the
> > segment could not be merged, that means the hbase tables could not be
> > cleaned. That`t terrible.Give me some advice?
> > Thank you! Looking forward to your reply.
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
Re: A problem about Global Dictionary and cube merge
Posted by ShaoFeng Shi <sh...@apache.org>.
I see; Did you use GlobalDictionaryBuilder for a dimension column? The GD
should only be used for count distinct measure, not suitable for dimension
encoding.
2017-09-28 23:57 GMT+08:00 柯南 <zz...@qq.com>:
> Thank you for your reply!
> I know this command.But Because its segment could not be merged, even if
> run this command ,the hbase tables could not be cleaned.
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年9月28日(星期四) 晚上9:29
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: A problem about Global Dictionary and cube merge
>
>
>
> You need run this:
> https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html
>
> 2017-09-28 20:40 GMT+08:00 柯南 <zz...@qq.com>:
>
> > hi,all：
> > Because of our yarn and hbase`s version limited,we deploy apache
> > kylin 1.6.0. For the correct result of 'count(distinct A_col)' across
> > segments, we must use Global Dictionary to A_col. When the cube contain
> > configuration of 'org.apache.kylin.dict.GlobalDictionaryBuilder', its
> > segment could not be merged.(AppendTrieDictionary class don`t support
> > getValueBytesFromIdImpl method and error is below) We will build the cube
> > every day, that means a new hbase table will be created every day. And
> the
> > segment could not be merged, that means the hbase tables could not be
> > cleaned. That`t terrible.Give me some advice?
> > Thank you! Looking forward to your reply.
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋