You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2017/12/07 09:40:09 UTC
Re: kylin streaming cube merge strategy?
Hi, Could you share the merge settings, and the logs when a new segment be
built? If not ready for auto merge, Kylin will print a message "Not ready
for merge on cube". Please provide detail log for troubleshooting.
2017-12-07 14:38 GMT+08:00 446463844@qq.com <44...@qq.com>:
> Hi:
> I try to use kylin streaming cube for building cube from kafka.
> And when I design the cube I set cube merge options on advanced Settings
> panel.
> when I build cube for times .I found the merge strategy not valid
> Can somebody help?
>
>
> 446463844@qq.com
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Re: kylin streaming cube merge strategy?
Posted by "446463844@qq.com" <44...@qq.com>.
Hi:
I use streaming cube for days .and auto build per 10 mintues
in monitor page I find no auto merge job is triggered.
And mine merge options be setting :
0.5 hour
1 hour
2 hour
4 hour
8 hour
1 day
7 day
Is there all most of segment not reached the merge section?
446463844@qq.com
From: ShaoFeng Shi
Date: 2017-12-08 16:08
To: dev
Subject: Re: kylin streaming cube merge strategy?
Correct; Its merge is not by offset, but by start/end time of the segments;
A timestamp field on message is required.
2017-12-08 15:51 GMT+08:00 funky <fu...@gmail.com>:
> thx,I try to understand it .Streaming cube build success and named with
> offset rather than timestamp means that it will be merged with internal
> timestamp (data structure must contains a 'timestamp' field in kafka).is
> that rigtht ?
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>
--
Best regards,
Shaofeng Shi 史少锋
Re: kylin streaming cube merge strategy?
Posted by ShaoFeng Shi <sh...@apache.org>.
Correct; Its merge is not by offset, but by start/end time of the segments;
A timestamp field on message is required.
2017-12-08 15:51 GMT+08:00 funky <fu...@gmail.com>:
> thx,I try to understand it .Streaming cube build success and named with
> offset rather than timestamp means that it will be merged with internal
> timestamp (data structure must contains a 'timestamp' field in kafka).is
> that rigtht ?
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>
--
Best regards,
Shaofeng Shi 史少锋
Re: kylin streaming cube merge strategy?
Posted by funky <fu...@gmail.com>.
thx,I try to understand it .Streaming cube build success and named with
offset rather than timestamp means that it will be merged with internal
timestamp (data structure must contains a 'timestamp' field in kafka).is
that rigtht ?
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: kylin streaming cube merge strategy?
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi,
The merge will check the start/end time of adjacent cube segments; If their
time has reached one of the merging thresholds, it will trigger a merge job.
For example, the min. and max. timestamp of 3 segments are:
seg1: 1:00 to 1:05
seg2: 1:05 to 1:20
seg3: 1:18 to 1:35
Assume you have defined 30 minutes as a threshold. Then when seg3 is built,
Kylin found seg1 to seg3 has reached this threshold, then it will merge
them. After the merge, the new segment is:
seg': 1:00 to 1:35
If you didn't see a merge be triggered, some possible situations:
1) there is hold among the segments; (the merging segments need be
continuous on Kafka)
2) the start/end time of each segment already exceeds some merging
threshold, but not reach a bigger threshold.
Please double check, or share more information for the investigation.
2017-12-08 13:33 GMT+08:00 funky <fu...@gmail.com>:
> er. I can manual merge the cube now,So can somebody tell me the streaming
> cube merge strategy automatic ?
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>
--
Best regards,
Shaofeng Shi 史少锋
Re: kylin streaming cube merge strategy?
Posted by funky <fu...@gmail.com>.
er. I can manual merge the cube now,So can somebody tell me the streaming
cube merge strategy automatic ?
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: kylin streaming cube merge strategy?
Posted by funky <fu...@gmail.com>.
yes I found "Ngt ready for merge on cube streaming_cube n cube " in kylin.log
I wonder that when I maunl merge cube segment in kylin Web UI and options
message that
Find no segments to merge by data range 1512518400000-1512604800000 for
CUBE[mate=streaming_version_cube]
the merge strategy when I design this cube
Auto Merge Thresholds
0.5hour
1hour
2hour
4hour
8hour
1day
7day
28days
--
Sent from: http://apache-kylin.74782.x6.nabble.com/