You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sendoh <un...@gmail.com> on 2016/11/28 12:24:22 UTC
How to let 1.1.3 not drop late events as verion 1.0.3 does
Hi Flink users,
Can I ask how to avoid default allowLateness(0) ? so that late events
becomes single-element windows as 1.0.3 version acts?
Best,
Sendoh
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by Sendoh <un...@gmail.com>.
Thanks, I follow your suggestion and it works as we expected.
Best,
Sendoh
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10510.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by vinay patil <vi...@gmail.com>.
Hi Aljoscha,
Yes that it true, but user who are new to flink does not know how to handle
the case where they cannot discard late data, so they end up writing custom
trigger which was not the case in 1.0.3
Regards,
Vinay Patil
On Tue, Nov 29, 2016 at 4:29 PM, Aljoscha Krettek [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Yes, but the allowedLateness() setting is that flag, no?
>
> On Tue, 29 Nov 2016 at 11:04 vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10364&i=0>> wrote:
>
>> Yes, That's what I have done.
>>
>> IMO flink should allow user to decide whether to discard late data or not
>> by having some flag, because for some cases users cannot afford to loose
>> any data.
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 29, 2016 at 3:16 PM, Maximilian Michels [via Apache Flink
>> User Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=10363&i=0>> wrote:
>> Setting allowedLateness to Long.MAX_VALUE and returning
>> TriggerResult.FIRE_AND_PURGE in the custom trigger should do the
>> trick.
>>
>> -Max
>>
>
>
> On Mon, Nov 28, 2016 at 2:57 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10362&i=0>> wrote:
>
> > Hi Sendoh,
> >
> > I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger,
> and
> > kept the allowedLateness value to Long.MAX_VALUE.
> > Because of this change the late elements are not discarded and become
> single
> > element windows
> >
> > Regards,
> > Vinay Patil
> >
> > On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via Apache Flink User Mailing
> List
> > archive.] <[hidden email]> wrote:
>
> >>
> >> Hi Flink users,
> >>
> >> Can I ask how to avoid default allowLateness(0) ? so that late events
> >> becomes single-element windows as 1.0.3 version acts?
> >>
> >> Best,
> >>
> >> Sendoh
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-
> verion-1-0-3-does-tp10349.html
> >> To start a new topic under Apache Flink User Mailing List archive.,
> email
> >> [hidden email]
> >> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> >> NAML
> >
> >
> >
> > ________________________________
> > View this message in context: Re: How to let 1.1.3 not drop late events
> as
> > verion 1.0.3 does
> >
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> > at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-
> tp10349p10362.html
> To start a new topic under Apache Flink User Mailing List archive., email [hidden
> email] <http:///user/SendEmail.jtp?type=node&node=10363&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> ------------------------------
> View this message in context: Re: How to let 1.1.3 not drop late events
> as verion 1.0.3 does
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10363.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-
> tp10349p10364.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10366.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, but the allowedLateness() setting is that flag, no?
On Tue, 29 Nov 2016 at 11:04 vinay patil <vi...@gmail.com> wrote:
> Yes, That's what I have done.
>
> IMO flink should allow user to decide whether to discard late data or not
> by having some flag, because for some cases users cannot afford to loose
> any data.
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 29, 2016 at 3:16 PM, Maximilian Michels [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10363&i=0>> wrote:
>
> Setting allowedLateness to Long.MAX_VALUE and returning
> TriggerResult.FIRE_AND_PURGE in the custom trigger should do the
> trick.
>
> -Max
>
>
>
> On Mon, Nov 28, 2016 at 2:57 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10362&i=0>> wrote:
>
> > Hi Sendoh,
> >
> > I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger,
> and
> > kept the allowedLateness value to Long.MAX_VALUE.
> > Because of this change the late elements are not discarded and become
> single
> > element windows
> >
> > Regards,
> > Vinay Patil
> >
> > On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via Apache Flink User Mailing
> List
> > archive.] <[hidden email]> wrote:
>
> >>
> >> Hi Flink users,
> >>
> >> Can I ask how to avoid default allowLateness(0) ? so that late events
> >> becomes single-element windows as 1.0.3 version acts?
> >>
> >> Best,
> >>
> >> Sendoh
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349.html
> >> To start a new topic under Apache Flink User Mailing List archive.,
> email
> >> [hidden email]
> >> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> >> NAML
> >
> >
> >
> > ________________________________
> > View this message in context: Re: How to let 1.1.3 not drop late events
> as
> > verion 1.0.3 does
> >
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> > at Nabble.com.
>
>
> ------------------------------
>
> If you reply to this email, your message will be added to the discussion
> below:
>
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10362.html
>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden
> email] <http:///user/SendEmail.jtp?type=node&node=10363&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> ------------------------------
> View this message in context: Re: How to let 1.1.3 not drop late events
> as verion 1.0.3 does
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10363.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by vinay patil <vi...@gmail.com>.
Yes, That's what I have done.
IMO flink should allow user to decide whether to discard late data or not
by having some flag, because for some cases users cannot afford to loose
any data.
Regards,
Vinay Patil
On Tue, Nov 29, 2016 at 3:16 PM, Maximilian Michels [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Setting allowedLateness to Long.MAX_VALUE and returning
> TriggerResult.FIRE_AND_PURGE in the custom trigger should do the
> trick.
>
> -Max
>
>
> On Mon, Nov 28, 2016 at 2:57 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10362&i=0>> wrote:
>
> > Hi Sendoh,
> >
> > I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger,
> and
> > kept the allowedLateness value to Long.MAX_VALUE.
> > Because of this change the late elements are not discarded and become
> single
> > element windows
> >
> > Regards,
> > Vinay Patil
> >
> > On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via Apache Flink User Mailing
> List
> > archive.] <[hidden email]> wrote:
> >>
> >> Hi Flink users,
> >>
> >> Can I ask how to avoid default allowLateness(0) ? so that late events
> >> becomes single-element windows as 1.0.3 version acts?
> >>
> >> Best,
> >>
> >> Sendoh
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-
> verion-1-0-3-does-tp10349.html
> >> To start a new topic under Apache Flink User Mailing List archive.,
> email
> >> [hidden email]
> >> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> >> NAML
> >
> >
> >
> > ________________________________
> > View this message in context: Re: How to let 1.1.3 not drop late events
> as
> > verion 1.0.3 does
> >
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> > at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-
> tp10349p10362.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10363.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by Maximilian Michels <mx...@apache.org>.
Setting allowedLateness to Long.MAX_VALUE and returning
TriggerResult.FIRE_AND_PURGE in the custom trigger should do the
trick.
-Max
On Mon, Nov 28, 2016 at 2:57 PM, vinay patil <vi...@gmail.com> wrote:
> Hi Sendoh,
>
> I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger, and
> kept the allowedLateness value to Long.MAX_VALUE.
> Because of this change the late elements are not discarded and become single
> element windows
>
> Regards,
> Vinay Patil
>
> On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via Apache Flink User Mailing List
> archive.] <[hidden email]> wrote:
>>
>> Hi Flink users,
>>
>> Can I ask how to avoid default allowLateness(0) ? so that late events
>> becomes single-element windows as 1.0.3 version acts?
>>
>> Best,
>>
>> Sendoh
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> [hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>
>
>
> ________________________________
> View this message in context: Re: How to let 1.1.3 not drop late events as
> verion 1.0.3 does
>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
Re: How to let 1.1.3 not drop late events as verion 1.0.3 does
Posted by vinay patil <vi...@gmail.com>.
Hi Sendoh,
I have used the Custom Trigger which is same as 1.0.3 EventTimeTrigger, and
kept the allowedLateness value to Long.MAX_VALUE.
Because of this change the late elements are not discarded and become
single element windows
Regards,
Vinay Patil
On Mon, Nov 28, 2016 at 5:54 PM, Sendoh [via Apache Flink User Mailing List
archive.] <ml...@n4.nabble.com> wrote:
> Hi Flink users,
>
> Can I ask how to avoid default allowLateness(0) ? so that late events
> becomes single-element windows as 1.0.3 version acts?
>
> Best,
>
> Sendoh
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-
> verion-1-0-3-does-tp10349.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-let-1-1-3-not-drop-late-events-as-verion-1-0-3-does-tp10349p10350.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.