You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 王文辉 <wa...@foxmail.com> on 2019/12/09 06:05:10 UTC
kylin构建的cube无法自动合并
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
回复： kylin构建的cube无法自动合并
Posted by 王文辉 <wa...@foxmail.com>.
sorry, I can't change my kylin version casually, because I need to consider compatibility with other big data component versions
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2019年12月9日(星期一) 下午3:59
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi:
I found that the auto merge feature have a issue on kylin 2.2.0:
https://issues.apache.org/jira/browse/KYLIN-3165
And this issue have been resolved in kylin 2.3.0.
If you want to use the feature of the auto merge, you can upgrade to 2.3.0 or the latest version.
在 2019年12月9日，15:49，王文辉 <wang_wen_hui@foxmail.com> 写道：
kylin version: 2.2.0
hadoop version: 2.6.0
CDH: 5.8.0
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2019年12月9日(星期一) 下午3:41
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi wenhui:
It’s a strange problem.
Can you provide your Kylin version and Hadoop enviornment? I will try to reproduce this.
在 2019年12月9日，14:59，王文辉 <wang_wen_hui@foxmail.com> 写道：
I know your meaning ,the number of READY segments met the conditions of auto merge, but auto merge was not triggered, the picture as follows:
<3A308EFA@3A718E5D.D8F0ED5D.jpg.jpg>
my poverty as follows:
<5D2F8E90@9444D246.D8F0ED5D.jpg.jpg>
------------------ 原始邮件 ------------------
发件人: "ztp"<ztp@apache.org>;
发送时间: 2019年12月9日(星期一) 下午2:32
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi Wenhui,
Only if your number of segments exceed the Auto Merge Thresholds, the independent segments will be merged.
For example: there are 3 segments originally and you build the cube today, so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be merged.
If you really do not want there are more segments, you can adjust the Auto Merge Thresholds to 1 or 2 days.
BTW: The Retention Threshold may be misleading, it is used to guarantee the recent xx days segments will be not merged as the recent segments may be refreshed separately.
On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wang_wen_hui@foxmail.com> wrote:
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
Re: kylin构建的cube无法自动合并
Posted by Yaqian Zhang <Ya...@126.com>.
Hi:
I found that the auto merge feature have a issue on kylin 2.2.0:
https://issues.apache.org/jira/browse/KYLIN-3165 <https://issues.apache.org/jira/browse/KYLIN-3165>
And this issue have been resolved in kylin 2.3.0.
If you want to use the feature of the auto merge, you can upgrade to 2.3.0 or the latest version.
> 在 2019年12月9日，15:49，王文辉 <wa...@foxmail.com> 写道：
>
> kylin version: 2.2.0
> hadoop version: 2.6.0
> CDH: 5.8.0
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Yaqian Zhang"<Ya...@126.com>;
> 发送时间: 2019年12月9日(星期一) 下午3:41
> 收件人: "user"<us...@kylin.apache.org>;
> 主题: Re: kylin构建的cube无法自动合并
>
> Hi wenhui:
>
> It’s a strange problem.
>
> Can you provide your Kylin version and Hadoop enviornment? I will try to reproduce this.
>
>> 在 2019年12月9日，14:59，王文辉 <wang_wen_hui@foxmail.com <ma...@foxmail.com>> 写道：
>>
>> I know your meaning ,the number of READY segments met the conditions of auto merge, but auto merge was not triggered, the picture as follows:
>> <3A...@3A718E5D.D8F0ED5D.jpg.jpg>
>> my poverty as follows:
>> <5D...@9444D246.D8F0ED5D.jpg.jpg>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "ztp"<ztp@apache.org <ma...@apache.org>>;
>> 发送时间: 2019年12月9日(星期一) 下午2:32
>> 收件人: "user"<user@kylin.apache.org <ma...@kylin.apache.org>>;
>> 主题: Re: kylin构建的cube无法自动合并
>>
>>
>> Hi Wenhui,
>>
>> Only if your number of segments exceed the Auto Merge Thresholds, the independent segments will be merged.
>>
>> For example: there are 3 segments originally and you build the cube today, so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be merged.
>>
>> If you really do not want there are more segments, you can adjust the Auto Merge Thresholds to 1 or 2 days.
>>
>> BTW: The Retention Threshold may be misleading, it is used to guarantee the recent xx days segments will be not merged as the recent segments may be refreshed separately.
>>
>> On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wang_wen_hui@foxmail.com <ma...@foxmail.com>> wrote:
>> 你好：
>> 我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
>> Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
>> 我在这三个中均填写了kylin.cube.is <http://kylin.cube.is/>-automerge-enabled 为 true
>> 还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
>> 以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
>>
>
回复： kylin构建的cube无法自动合并
Posted by 王文辉 <wa...@foxmail.com>.
kylin version: 2.2.0
hadoop version: 2.6.0
CDH: 5.8.0
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2019年12月9日(星期一) 下午3:41
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi wenhui:
It’s a strange problem.
Can you provide your Kylin version and Hadoop enviornment? I will try to reproduce this.
在 2019年12月9日，14:59，王文辉 <wang_wen_hui@foxmail.com> 写道：
I know your meaning ,the number of READY segments met the conditions of auto merge, but auto merge was not triggered, the picture as follows:
<3A308EFA@3A718E5D.D8F0ED5D.jpg.jpg>
my poverty as follows:
<5D2F8E90@9444D246.D8F0ED5D.jpg.jpg>
------------------ 原始邮件 ------------------
发件人: "ztp"<ztp@apache.org>;
发送时间: 2019年12月9日(星期一) 下午2:32
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi Wenhui,
Only if your number of segments exceed the Auto Merge Thresholds, the independent segments will be merged.
For example: there are 3 segments originally and you build the cube today, so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be merged.
If you really do not want there are more segments, you can adjust the Auto Merge Thresholds to 1 or 2 days.
BTW: The Retention Threshold may be misleading, it is used to guarantee the recent xx days segments will be not merged as the recent segments may be refreshed separately.
On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wang_wen_hui@foxmail.com> wrote:
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
Re: kylin构建的cube无法自动合并
Posted by Yaqian Zhang <Ya...@126.com>.
Hi wenhui:
It’s a strange problem.
Can you provide your Kylin version and Hadoop enviornment? I will try to reproduce this.
> 在 2019年12月9日，14:59，王文辉 <wa...@foxmail.com> 写道：
>
> I know your meaning ,the number of READY segments met the conditions of auto merge, but auto merge was not triggered, the picture as follows:
> <3A...@3A718E5D.D8F0ED5D.jpg.jpg>
> my poverty as follows:
> <5D...@9444D246.D8F0ED5D.jpg.jpg>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ztp"<zt...@apache.org>;
> 发送时间: 2019年12月9日(星期一) 下午2:32
> 收件人: "user"<us...@kylin.apache.org>;
> 主题: Re: kylin构建的cube无法自动合并
>
>
> Hi Wenhui,
>
> Only if your number of segments exceed the Auto Merge Thresholds, the independent segments will be merged.
>
> For example: there are 3 segments originally and you build the cube today, so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be merged.
>
> If you really do not want there are more segments, you can adjust the Auto Merge Thresholds to 1 or 2 days.
>
> BTW: The Retention Threshold may be misleading, it is used to guarantee the recent xx days segments will be not merged as the recent segments may be refreshed separately.
>
> On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wang_wen_hui@foxmail.com <ma...@foxmail.com>> wrote:
> 你好：
> 我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
> Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
> 我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
> 还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
> 以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
>
回复： kylin构建的cube无法自动合并
Posted by 王文辉 <wa...@foxmail.com>.
I know your meaning ,the number of READY segments met the conditions of auto merge, but auto merge was not triggered, the picture as follows:
my poverty as follows:
------------------ 原始邮件 ------------------
发件人: "ztp"<ztp@apache.org>;
发送时间: 2019年12月9日(星期一) 下午2:32
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi Wenhui,
Only if your number of segments exceed the Auto Merge Thresholds, the independent segments will be merged.
For example: there are 3 segments originally and you build the cube today, so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be merged.
If you really do not want there are more segments, you can adjust the Auto Merge Thresholds to 1 or 2 days.
BTW: The Retention Threshold may be misleading, it is used to guarantee the recent xx days segments will be not merged as the recent segments may be refreshed separately.
On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wang_wen_hui@foxmail.com> wrote:
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
Re: kylin构建的cube无法自动合并
Posted by ztp <zt...@apache.org>.
Hi Wenhui,
Only if your number of segments exceed the Auto Merge Thresholds,
the independent segments will be merged.
For example: there are 3 segments originally and you build the cube today,
so, there will be 4 segments totally. But 4 < 7, the 4 segments will not be
merged.
If you really do not want there are more segments, you can adjust the Auto
Merge Thresholds to 1 or 2 days.
BTW: The Retention Threshold may be misleading, it is used to guarantee the
recent xx days segments will be not merged as the recent segments may be
refreshed separately.
On Mon, Dec 9, 2019 at 2:05 PM 王文辉 <wa...@foxmail.com> wrote:
> 你好：
> 我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
> Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
> 我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
> 还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention
> Threshold为 0
> 以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
>
>
回复： kylin构建的cube无法自动合并
Posted by 王文辉 <wa...@foxmail.com>.
------------------ 原始邮件 ------------------
发件人: "王文辉"<wang_wen_hui@foxmail.com>;
发送时间: 2019年12月9日(星期一) 下午2:39
收件人: "user"<user@kylin.apache.org>;
主题: 回复： kylin构建的cube无法自动合并
yes, the number of READY segments met the conditions of auto merge,but auto merge was not triggered.I don't why ?
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2019年12月9日(星期一) 下午2:35
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi:
The mechanism of auto-merge is as follows:
If your READY segment time range reaches 7 days but does not reach 28, the first step of automatic merging will be triggered, and the 7-day segment will be merged into a segment; if the range reach 28 days further, the 28 day segment will be merged into a segment.
You will still see one segment being built per day，until the segment time range reaches 7 days and the auto-merge task be triggered.
Has the number of READY segments met the conditions of auto merge and auto merge was not triggered?
在 2019年12月9日，14:05，王文辉 <wang_wen_hui@foxmail.com> 写道：
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
回复： kylin构建的cube无法自动合并
Posted by 王文辉 <wa...@foxmail.com>.
yes, the number of READY segments met the conditions of auto merge,but auto merge was not triggered.I don't why ?
------------------ 原始邮件 ------------------
发件人: "Yaqian Zhang"<Yaqian_Zhang@126.com>;
发送时间: 2019年12月9日(星期一) 下午2:35
收件人: "user"<user@kylin.apache.org>;
主题: Re: kylin构建的cube无法自动合并
Hi:
The mechanism of auto-merge is as follows:
If your READY segment time range reaches 7 days but does not reach 28, the first step of automatic merging will be triggered, and the 7-day segment will be merged into a segment; if the range reach 28 days further, the 28 day segment will be merged into a segment.
You will still see one segment being built per day，until the segment time range reaches 7 days and the auto-merge task be triggered.
Has the number of READY segments met the conditions of auto merge and auto merge was not triggered?
在 2019年12月9日，14:05，王文辉 <wang_wen_hui@foxmail.com> 写道：
你好：
我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
Re: kylin构建的cube无法自动合并
Posted by Yaqian Zhang <Ya...@126.com>.
Hi:
The mechanism of auto-merge is as follows:
If your READY segment time range reaches 7 days but does not reach 28, the first step of automatic merging will be triggered, and the 7-day segment will be merged into a segment; if the range reach 28 days further, the 28 day segment will be merged into a segment.
You will still see one segment being built per day，until the segment time range reaches 7 days and the auto-merge task be triggered.
Has the number of READY segments met the conditions of auto merge and auto merge was not triggered?
> 在 2019年12月9日，14:05，王文辉 <wa...@foxmail.com> 写道：
>
> 你好：
> 我构建的cube任务配置了自动合并操作，但却并不生效，其过程如下：
> Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件
> 我在这三个中均填写了kylin.cube.is-automerge-enabled 为 true
> 还有构建cube在Refresh Setting中，填写了Auto Merge Thresholds 为 7 Days, 28 Days，Retention Threshold为 0
> 以上操作均不生效，我的cube定时任务为每天执行一次，Hbase中仍然是每天一个segments
>