You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by hongbin ma <ma...@apache.org> on 2015/08/03 05:59:48 UTC
How to deal with empty segment
Do we have any resolution on how to deal with segment with 0 records now?
I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it is
closed now,
but i'm not aware of this by far.
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: How to deal with empty segment
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
+1; Kylin couldn’t differentiate whether the empty segment is expected or
is something wrong; Input data quality should be ensured by user before
building into cube; So, in my opinion, a warning message in the
notification of build job is good;
On 8/6/15, 10:37 PM, "周千昊" <z....@gmail.com> wrote:
>I think the such flag like 'AllowEmptySegment' should not be mixed up with
>cube or streaming metadata.
>It makes sense kylin allow empty segment by default and inform the user
>when it happens.
>Anyway our users' thoughts will be more helpful.
>Looking forward to seeing more comments
>
>Luke Han <lu...@gmail.com>于2015年8月6日周四 下午9:49写道：
>
>> Streaming & Batch are different behaviour, that's why we should leave
>>this
>> option to user
>> to allow them choose which exactly result they want.
>>
>> Add one setting option for each cube/streaming say "AllowEmptySegment",
>> default to be true
>>
>> Then when cubing engine could raise error when it set to false, and
>> continue if true.
>>
>> Even in streaming case, some of them are ok to merge empty later but
>>some
>> need alert.
>>
>> Thanks.
>>
>>
>> Best Regards!
>> ---------------------
>>
>> Luke Han
>>
>> On Thu, Aug 6, 2015 at 9:08 PM, hongbin ma <ma...@apache.org> wrote:
>>
>> > how about in the streaming case? even if we label a segment as
>>warning,
>> it
>> > might get merged before we notice it.
>> >
>> > On Thu, Aug 6, 2015 at 5:51 PM, Li Yang <li...@apache.org> wrote:
>> >
>> > > Luke and me discussed this once. I remember the conclusion is let
>>the
>> > build
>> > > pass, but leave a warn on GUI to remind user there might be error
>>about
>> > the
>> > > input source.
>> > >
>> > > On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org>
>> > wrote:
>> > >
>> > > > Do we have any resolution on how to deal with segment with 0
>>records
>> > now?
>> > > >
>> > > > I found a ticket https://issues.apache.org/jira/browse/KYLIN-345,
>>it
>> > is
>> > > > closed now,
>> > > > but i'm not aware of this by far.
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > > *Bin Mahone | 马洪宾*
>> > > > Apache Kylin: http://kylin.io
>> > > > Github: https://github.com/binmahone
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > *Bin Mahone | 马洪宾*
>> > Apache Kylin: http://kylin.io
>> > Github: https://github.com/binmahone
>> >
>>
>--
>Best Regard
>ZhouQianhao
Re: How to deal with empty segment
Posted by 周千昊 <z....@gmail.com>.
I think the such flag like 'AllowEmptySegment' should not be mixed up with
cube or streaming metadata.
It makes sense kylin allow empty segment by default and inform the user
when it happens.
Anyway our users' thoughts will be more helpful.
Looking forward to seeing more comments
Luke Han <lu...@gmail.com>于2015年8月6日周四 下午9:49写道：
> Streaming & Batch are different behaviour, that's why we should leave this
> option to user
> to allow them choose which exactly result they want.
>
> Add one setting option for each cube/streaming say "AllowEmptySegment",
> default to be true
>
> Then when cubing engine could raise error when it set to false, and
> continue if true.
>
> Even in streaming case, some of them are ok to merge empty later but some
> need alert.
>
> Thanks.
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Thu, Aug 6, 2015 at 9:08 PM, hongbin ma <ma...@apache.org> wrote:
>
> > how about in the streaming case? even if we label a segment as warning,
> it
> > might get merged before we notice it.
> >
> > On Thu, Aug 6, 2015 at 5:51 PM, Li Yang <li...@apache.org> wrote:
> >
> > > Luke and me discussed this once. I remember the conclusion is let the
> > build
> > > pass, but leave a warn on GUI to remind user there might be error about
> > the
> > > input source.
> > >
> > > On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org>
> > wrote:
> > >
> > > > Do we have any resolution on how to deal with segment with 0 records
> > now?
> > > >
> > > > I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it
> > is
> > > > closed now,
> > > > but i'm not aware of this by far.
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > *Bin Mahone | 马洪宾*
> > > > Apache Kylin: http://kylin.io
> > > > Github: https://github.com/binmahone
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
--
Best Regard
ZhouQianhao
Re: How to deal with empty segment
Posted by Luke Han <lu...@gmail.com>.
Streaming & Batch are different behaviour, that's why we should leave this
option to user
to allow them choose which exactly result they want.
Add one setting option for each cube/streaming say "AllowEmptySegment",
default to be true
Then when cubing engine could raise error when it set to false, and
continue if true.
Even in streaming case, some of them are ok to merge empty later but some
need alert.
Thanks.
Best Regards!
---------------------
Luke Han
On Thu, Aug 6, 2015 at 9:08 PM, hongbin ma <ma...@apache.org> wrote:
> how about in the streaming case? even if we label a segment as warning, it
> might get merged before we notice it.
>
> On Thu, Aug 6, 2015 at 5:51 PM, Li Yang <li...@apache.org> wrote:
>
> > Luke and me discussed this once. I remember the conclusion is let the
> build
> > pass, but leave a warn on GUI to remind user there might be error about
> the
> > input source.
> >
> > On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org>
> wrote:
> >
> > > Do we have any resolution on how to deal with segment with 0 records
> now?
> > >
> > > I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it
> is
> > > closed now,
> > > but i'm not aware of this by far.
> > >
> > > --
> > > Regards,
> > >
> > > *Bin Mahone | 马洪宾*
> > > Apache Kylin: http://kylin.io
> > > Github: https://github.com/binmahone
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>
Re: How to deal with empty segment
Posted by hongbin ma <ma...@apache.org>.
how about in the streaming case? even if we label a segment as warning, it
might get merged before we notice it.
On Thu, Aug 6, 2015 at 5:51 PM, Li Yang <li...@apache.org> wrote:
> Luke and me discussed this once. I remember the conclusion is let the build
> pass, but leave a warn on GUI to remind user there might be error about the
> input source.
>
> On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org> wrote:
>
> > Do we have any resolution on how to deal with segment with 0 records now?
> >
> > I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it is
> > closed now,
> > but i'm not aware of this by far.
> >
> > --
> > Regards,
> >
> > *Bin Mahone | 马洪宾*
> > Apache Kylin: http://kylin.io
> > Github: https://github.com/binmahone
> >
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: How to deal with empty segment
Posted by Luke Han <lu...@gmail.com>.
Then let's make sure there's alert message will be record somewhere,
and send mail out.
And I'm thinking about to bring this story from backlog to priority:
*Notify for announcement and notification:*
https://issues.apache.org/jira/browse/KYLIN-413
Each notification, alert and message will go to right place,
users, admin could easily aware such information.
Screenshot will looks like:
Thanks.
Best Regards!
---------------------
Luke Han
On Mon, Aug 10, 2015 at 11:21 AM, Li Yang <li...@apache.org> wrote:
> So the consensus is let empty segment pass. Great.
>
> On Mon, Aug 10, 2015 at 10:28 AM, Shi, Shaofeng <sh...@ebay.com> wrote:
>
> > As I remembered, the conclusion is, in the notification on job complete,
> > showing the source record size; From it user will know whether it is an
> > empty segment or not;
> >
> > While, due to some code level reason (need recall that), this message
> > hasn’t been added so far;
> >
> > On 8/6/15, 5:51 PM, "Li Yang" <li...@apache.org> wrote:
> >
> > >Luke and me discussed this once. I remember the conclusion is let the
> > >build
> > >pass, but leave a warn on GUI to remind user there might be error about
> > >the
> > >input source.
> > >
> > >On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org>
> wrote:
> > >
> > >> Do we have any resolution on how to deal with segment with 0 records
> > >>now?
> > >>
> > >> I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it
> is
> > >> closed now,
> > >> but i'm not aware of this by far.
> > >>
> > >> --
> > >> Regards,
> > >>
> > >> *Bin Mahone | 马洪宾*
> > >> Apache Kylin: http://kylin.io
> > >> Github: https://github.com/binmahone
> > >>
> >
> >
>
Re: How to deal with empty segment
Posted by Li Yang <li...@apache.org>.
So the consensus is let empty segment pass. Great.
On Mon, Aug 10, 2015 at 10:28 AM, Shi, Shaofeng <sh...@ebay.com> wrote:
> As I remembered, the conclusion is, in the notification on job complete,
> showing the source record size; From it user will know whether it is an
> empty segment or not;
>
> While, due to some code level reason (need recall that), this message
> hasn’t been added so far;
>
> On 8/6/15, 5:51 PM, "Li Yang" <li...@apache.org> wrote:
>
> >Luke and me discussed this once. I remember the conclusion is let the
> >build
> >pass, but leave a warn on GUI to remind user there might be error about
> >the
> >input source.
> >
> >On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org> wrote:
> >
> >> Do we have any resolution on how to deal with segment with 0 records
> >>now?
> >>
> >> I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it is
> >> closed now,
> >> but i'm not aware of this by far.
> >>
> >> --
> >> Regards,
> >>
> >> *Bin Mahone | 马洪宾*
> >> Apache Kylin: http://kylin.io
> >> Github: https://github.com/binmahone
> >>
>
>
Re: How to deal with empty segment
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
As I remembered, the conclusion is, in the notification on job complete,
showing the source record size; From it user will know whether it is an
empty segment or not;
While, due to some code level reason (need recall that), this message
hasn’t been added so far;
On 8/6/15, 5:51 PM, "Li Yang" <li...@apache.org> wrote:
>Luke and me discussed this once. I remember the conclusion is let the
>build
>pass, but leave a warn on GUI to remind user there might be error about
>the
>input source.
>
>On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org> wrote:
>
>> Do we have any resolution on how to deal with segment with 0 records
>>now?
>>
>> I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it is
>> closed now,
>> but i'm not aware of this by far.
>>
>> --
>> Regards,
>>
>> *Bin Mahone | 马洪宾*
>> Apache Kylin: http://kylin.io
>> Github: https://github.com/binmahone
>>
Re: How to deal with empty segment
Posted by Li Yang <li...@apache.org>.
Luke and me discussed this once. I remember the conclusion is let the build
pass, but leave a warn on GUI to remind user there might be error about the
input source.
On Mon, Aug 3, 2015 at 11:59 AM, hongbin ma <ma...@apache.org> wrote:
> Do we have any resolution on how to deal with segment with 0 records now?
>
> I found a ticket https://issues.apache.org/jira/browse/KYLIN-345, it is
> closed now,
> but i'm not aware of this by far.
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>