You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vinay Patil <vi...@gmail.com> on 2016/10/03 03:21:34 UTC
Regarding Late Elements
Hi Guys,
Just wanted to get an idea on Why Flink decided to completely discard late
elements in the latest version ?, this was not the case in 1.0.3
P.S In our case the data is critical so we cannot discard a single record
even if it is late, I have written a custom trigger (as suggested by
Aljoscha) to even accept late elements.
Regards,
Vinay Patil
Re: Regarding Late Elements
Posted by vinay patil <vi...@gmail.com>.
Yes Kostas, thank you for the explanation , I will take a look
Regards,
Vinay Patil
On Tue, Oct 4, 2016 at 11:23 AM, Kostas Kloudas [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi Vinay,
>
> These methods are useful when using your trigger with SessionWindows. When
> using session windows,
> the state of a window and that of the corresponding trigger has to be
> merged with that of other windows.
>
> These methods do exactly that: the canMerge() says if the trigger can be
> used with Session Windows, and if yes,
> the onMerge() includes the logic of what to do with the trigger state when
> a window merging happens.
>
> As an example you can check out the CountTrigger.
>
> I hope this answers your question.
>
> Cheers,
> Kostas
>
> On Oct 4, 2016, at 6:10 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9332&i=0>> wrote:
>
> Hi Kostas,
>
> Yes you are right , I am always doing FIRE_AND_PURGE , if we don't do
> this and only use FIRE , the window function will get the elements
> in incremental fashion (1, 2,3..so on)
> I had observed this while testing.
>
> Can you please explain me the importance of canMerge and onMerge functions
> in trigger, from the javadocs I got it on a high level, but not able to
> understand it correctly (so I am not using these functions for now in
> Custom Trigger).
>
> Regards,
> Vinay Patil
>
> On Tue, Oct 4, 2016 at 10:51 AM, Kostas Kloudas [via Apache Flink User
> Mailing List archive.] <<a href="x-msg://14/user/
> SendEmail.jtp?type=node&node=9331&i=0" target="_top"
> rel="nofollow" link="external" class="">[hidden email]> wrote:
>
>> Hi Vinay,
>>
>> By setting the allowed_lateness to LongMax you are ok.
>> Sorry I forgot that this was the default value.
>>
>> Just a note (although you have it right in your code), in this case
>> you should always FIRE_AND_PURGE and not just FIRE. In other
>> case your state will keep growing as it is never garbage collected.
>>
>> Cheers,
>> Kostas
>>
>>
>> On Oct 4, 2016, at 5:42 PM, vinay patil <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9329&i=0>> wrote:
>>
>> Long.MAX_VALUE
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Regarding-Late-Elements-tp9284p9329.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> <a href="x-msg://14/user/SendEmail.jtp?type=node&node=9331&i=1"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Regarding Late Elements
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9331.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com <http://nabble.com>.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9332.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9335.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Vinay,
These methods are useful when using your trigger with SessionWindows. When using session windows,
the state of a window and that of the corresponding trigger has to be merged with that of other windows.
These methods do exactly that: the canMerge() says if the trigger can be used with Session Windows, and if yes,
the onMerge() includes the logic of what to do with the trigger state when a window merging happens.
As an example you can check out the CountTrigger.
I hope this answers your question.
Cheers,
Kostas
> On Oct 4, 2016, at 6:10 PM, vinay patil <vi...@gmail.com> wrote:
>
> Hi Kostas,
>
> Yes you are right , I am always doing FIRE_AND_PURGE , if we don't do this and only use FIRE , the window function will get the elements in incremental fashion (1, 2,3..so on)
> I had observed this while testing.
>
> Can you please explain me the importance of canMerge and onMerge functions in trigger, from the javadocs I got it on a high level, but not able to understand it correctly (so I am not using these functions for now in Custom Trigger).
>
> Regards,
> Vinay Patil
>
> On Tue, Oct 4, 2016 at 10:51 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] <[hidden email] <x-msg://14/user/SendEmail.jtp?type=node&node=9331&i=0>> wrote:
> Hi Vinay,
>
> By setting the allowed_lateness to LongMax you are ok.
> Sorry I forgot that this was the default value.
>
> Just a note (although you have it right in your code), in this case
> you should always FIRE_AND_PURGE and not just FIRE. In other
> case your state will keep growing as it is never garbage collected.
>
> Cheers,
> Kostas
>
>
>> On Oct 4, 2016, at 5:42 PM, vinay patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=9329&i=0>> wrote:
>>
>> Long.MAX_VALUE
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9329.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9329.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <x-msg://14/user/SendEmail.jtp?type=node&node=9331&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://6844F060-0841-49DA-811F-B7AE5E851424>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9331.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com.
Re: Regarding Late Elements
Posted by vinay patil <vi...@gmail.com>.
Hi Kostas,
Yes you are right , I am always doing FIRE_AND_PURGE , if we don't do this
and only use FIRE , the window function will get the elements
in incremental fashion (1, 2,3..so on)
I had observed this while testing.
Can you please explain me the importance of canMerge and onMerge functions
in trigger, from the javadocs I got it on a high level, but not able to
understand it correctly (so I am not using these functions for now in
Custom Trigger).
Regards,
Vinay Patil
On Tue, Oct 4, 2016 at 10:51 AM, Kostas Kloudas [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi Vinay,
>
> By setting the allowed_lateness to LongMax you are ok.
> Sorry I forgot that this was the default value.
>
> Just a note (although you have it right in your code), in this case
> you should always FIRE_AND_PURGE and not just FIRE. In other
> case your state will keep growing as it is never garbage collected.
>
> Cheers,
> Kostas
>
>
> On Oct 4, 2016, at 5:42 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9329&i=0>> wrote:
>
> Long.MAX_VALUE
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9329.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9331.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Vinay,
By setting the allowed_lateness to LongMax you are ok.
Sorry I forgot that this was the default value.
Just a note (although you have it right in your code), in this case
you should always FIRE_AND_PURGE and not just FIRE. In other
case your state will keep growing as it is never garbage collected.
Cheers,
Kostas
> On Oct 4, 2016, at 5:42 PM, vinay patil <vi...@gmail.com> wrote:
>
> Long.MAX_VALUE
Re: Regarding Late Elements
Posted by vinay patil <vi...@gmail.com>.
Hi Kostas,
The late elements are immediately getting triggered with the code I have
sent,
I have tested it with a test case as follows : (I am doing the outer-join
operation by doing the union of stream1 and stream2)
1. Push 5 records to Kafka Topic 1 -----> sourceStream1
2. Wait for few minutes -- (Now by this time sourceStream1 elements are
triggered as non-matched elements)
3. Push the matching 5 records to Kafka Topic 2 ------> sourceStream2 (here
the watermark has already reached ahead)
Now according the custom trigger whenever *onElement is called it
immediately fires the window and purge it*
I have kept the onEventTime code same as 1.0.3, where it will fire and purge
Am I doing something wrong ? (I have kept allowedLateness to Long.MAX_VALUE)
Regards,
Vinay Patil
On Tue, Oct 4, 2016 at 9:58 AM, Kostas Kloudas [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hi Vinay,
>
> From what I understand from your code, the only difference of your trigger
> compared to the
> one shipping with Flink is that for the late elements, instead of firing
> and keeping the element,
> you fire and purge, i.e. clean the window state.
>
> This does not solve the problem of dropping the super late elements if
> their window has expired
> (currentWatermark >= window.maxTimestamp + allowedLateness).
>
> Cheers,
> Kostas
>
> On Oct 4, 2016, at 3:25 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9325&i=0>> wrote:
>
> Hi Kostas,
>
> Thank you for your reply, yes that will be a good functionality to have,
> but for now the Custom Trigger as close to 1.0.3 works for me.
> public TriggerResult onElement(Object element, long timestamp, TimeWindow
> window, TriggerContext ctx) throws Exception { if(window.maxTimestamp()
> <= ctx.getCurrentWatermark()) { *return TriggerResult.FIRE_AND_PURGE;* }
> else { ctx.registerEventTimeTimer(window.maxTimestamp()); return
> TriggerResult.CONTINUE; } } public TriggerResult onEventTime(long time,
> TimeWindow window, TriggerContext ctx) { return
> *TriggerResult.FIRE_AND_PURGE;* }
>
> This is the change I have done in my custom trigger which is similar to
> 1.0.3 , does this looks good to you ?
> (I am not using canMerge and onMerge methods since I am not aware of it)
>
>
> Regards,
> Vinay Patil
>
> On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User
> Mailing List archive.] <<a href="x-msg://9/user/
> SendEmail.jtp?type=node&node=9323&i=0" target="_top"
> rel="nofollow" link="external" class="">[hidden email]> wrote:
>
>> Hello LF and Vinay,
>>
>> With the introduction of “allowed lateness” elements and windows are
>> kept around until the watermark
>> passes the window.maxTimestamp + allowed_lateness and then they are
>> cleaned up (garbage collected)
>>
>> Every element that comes in and belongs to a window that is garbage
>> collected is dropped as super-late.
>> Elements that are late, but no more than the allowed lateness, they are
>> kept the window fires as before.
>>
>> If you know what the maximum latency is, then the best way is to set the
>> allowed lateness to that value.
>>
>> Currently Flink drops super-late elements and does not provide any
>> mechanism to manually handle these
>> elements, BUT there are discussions about adding such a functionality
>> that will allow you to (probably) have
>> a separate stream with only these elements.
>>
>> Thanks,
>> Kostas
>>
>> On Oct 3, 2016, at 5:28 PM, [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9307&i=0> wrote:
>>
>> Not yet.
>> I'm hoping a Flink export on this mailing list will reply.
>>
>>
>> - LF
>>
>>
>>
>> ------------------------------
>> *From:* vinay patil <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=9307&i=1>>
>> *To:* [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=2>
>> *Sent:* Monday, October 3, 2016 8:09 AM
>> *Subject:* Re: Regarding Late Elements
>>
>> Hi LF,
>>
>> So did you manage to get the workaround for it ?
>>
>> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>>
>> Regards,
>> Vinay Patil
>>
>> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing
>> List archive.] <[hidden email]> wrote:
>>
>> We have the same requirement - we cannot discard any data even if it
>> arrives late.
>>
>>
>> - LF
>>
>>
>>
>>
>> ------------------------------
>> *From:* Vinay Patil <[hidden email]>
>> *To:* [hidden email]
>>
>> *Sent:* Sunday, October 2, 2016 8:21 PM
>> *Subject:* Regarding Late Elements
>>
>> Hi Guys,
>>
>> Just wanted to get an idea on Why Flink decided to completely discard
>> late elements in the latest version ?, this was not the case in 1.0.3
>>
>>
>> P.S In our case the data is critical so we cannot discard a single record
>> even if it is late, I have written a custom trigger (as suggested by
>> Aljoscha) to even accept late elements.
>>
>>
>> Regards,
>> Vinay Patil
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user- mailing-list-archive.2336050.
>> n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Regarding Late Elements
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com <http://nabble.com/>.
>>
>>
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Regarding-Late-Elements-tp9284p9307.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> <a href="x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=1"
>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Regarding Late Elements
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9323.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com <http://nabble.com>.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9325.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9328.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Vinay,
From what I understand from your code, the only difference of your trigger compared to the
one shipping with Flink is that for the late elements, instead of firing and keeping the element,
you fire and purge, i.e. clean the window state.
This does not solve the problem of dropping the super late elements if their window has expired
(currentWatermark >= window.maxTimestamp + allowedLateness).
Cheers,
Kostas
> On Oct 4, 2016, at 3:25 PM, vinay patil <vi...@gmail.com> wrote:
>
> Hi Kostas,
>
> Thank you for your reply, yes that will be a good functionality to have, but for now the Custom Trigger as close to 1.0.3 works for me.
>
>
> public TriggerResult onElement(Object element, long timestamp, TimeWindow window, TriggerContext ctx) throws Exception {
> if(window.maxTimestamp() <= ctx.getCurrentWatermark()) {
> return TriggerResult.FIRE_AND_PURGE;
> } else {
> ctx.registerEventTimeTimer(window.maxTimestamp());
> return TriggerResult.CONTINUE;
> }
> }
> public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) { return TriggerResult.FIRE_AND_PURGE; }
>
> This is the change I have done in my custom trigger which is similar to 1.0.3 , does this looks good to you ?
> (I am not using canMerge and onMerge methods since I am not aware of it)
>
>
> Regards,
> Vinay Patil
>
> On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User Mailing List archive.] <[hidden email] <x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=0>> wrote:
> Hello LF and Vinay,
>
> With the introduction of “allowed lateness” elements and windows are kept around until the watermark
> passes the window.maxTimestamp + allowed_lateness and then they are cleaned up (garbage collected)
>
> Every element that comes in and belongs to a window that is garbage collected is dropped as super-late.
> Elements that are late, but no more than the allowed lateness, they are kept the window fires as before.
>
> If you know what the maximum latency is, then the best way is to set the allowed lateness to that value.
>
> Currently Flink drops super-late elements and does not provide any mechanism to manually handle these
> elements, BUT there are discussions about adding such a functionality that will allow you to (probably) have
> a separate stream with only these elements.
>
> Thanks,
> Kostas
>
>> On Oct 3, 2016, at 5:28 PM, [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=0> wrote:
>>
>> Not yet.
>> I'm hoping a Flink export on this mailing list will reply.
>>
>>
>> - LF
>>
>>
>>
>> From: vinay patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=1>>
>> To: [hidden email] <http://user/SendEmail.jtp?type=node&node=9307&i=2>
>> Sent: Monday, October 3, 2016 8:09 AM
>> Subject: Re: Regarding Late Elements
>>
>> Hi LF,
>>
>> So did you manage to get the workaround for it ?
>>
>> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>>
>> Regards,
>> Vinay Patil
>>
>> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] <[hidden email] <>> wrote:
>> We have the same requirement - we cannot discard any data even if it arrives late.
>>
>>
>> - LF
>>
>>
>>
>>
>> From: Vinay Patil <[hidden email] <>>
>> To: [hidden email] <>
>>
>> Sent: Sunday, October 2, 2016 8:21 PM
>> Subject: Regarding Late Elements
>>
>> Hi Guys,
>>
>> Just wanted to get an idea on Why Flink decided to completely discard late elements in the latest version ?, this was not the case in 1.0.3
>>
>>
>> P.S In our case the data is critical so we cannot discard a single record even if it is late, I have written a custom trigger (as suggested by Aljoscha) to even accept late elements.
>>
>>
>> Regards,
>> Vinay Patil
>>
>>
>>
>>
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-flink-user- mailing-list-archive.2336050. n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <>
>> To unsubscribe from Apache Flink User Mailing List archive., click here <>.
>> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com <http://nabble.com/>.
>>
>>
>>
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9307.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9307.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <x-msg://9/user/SendEmail.jtp?type=node&node=9323&i=1>
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://73527FE2-B120-400C-8866-E2985C929827>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9323.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com.
Re: Regarding Late Elements
Posted by vinay patil <vi...@gmail.com>.
Hi Kostas,
Thank you for your reply, yes that will be a good functionality to have,
but for now the Custom Trigger as close to 1.0.3 works for me.
public TriggerResult onElement(Object element, long timestamp, TimeWindow
window, TriggerContext ctx) throws Exception { if(window.maxTimestamp() <=
ctx.getCurrentWatermark()) { *return TriggerResult.FIRE_AND_PURGE;* } else
{ ctx.registerEventTimeTimer(window.maxTimestamp()); return
TriggerResult.CONTINUE; } } public TriggerResult onEventTime(long time,
TimeWindow window, TriggerContext ctx) { return
*TriggerResult.FIRE_AND_PURGE;* }
This is the change I have done in my custom trigger which is similar to
1.0.3 , does this looks good to you ?
(I am not using canMerge and onMerge methods since I am not aware of it)
Regards,
Vinay Patil
On Tue, Oct 4, 2016 at 3:58 AM, Kostas Kloudas [via Apache Flink User
Mailing List archive.] <ml...@n4.nabble.com> wrote:
> Hello LF and Vinay,
>
> With the introduction of “allowed lateness” elements and windows are kept
> around until the watermark
> passes the window.maxTimestamp + allowed_lateness and then they are
> cleaned up (garbage collected)
>
> Every element that comes in and belongs to a window that is garbage
> collected is dropped as super-late.
> Elements that are late, but no more than the allowed lateness, they are
> kept the window fires as before.
>
> If you know what the maximum latency is, then the best way is to set the
> allowed lateness to that value.
>
> Currently Flink drops super-late elements and does not provide any
> mechanism to manually handle these
> elements, BUT there are discussions about adding such a functionality that
> will allow you to (probably) have
> a separate stream with only these elements.
>
> Thanks,
> Kostas
>
> On Oct 3, 2016, at 5:28 PM, [hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9307&i=0> wrote:
>
> Not yet.
> I'm hoping a Flink export on this mailing list will reply.
>
>
> - LF
>
>
>
> ------------------------------
> *From:* vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9307&i=1>>
> *To:* [hidden email] <http:///user/SendEmail.jtp?type=node&node=9307&i=2>
> *Sent:* Monday, October 3, 2016 8:09 AM
> *Subject:* Re: Regarding Late Elements
>
> Hi LF,
>
> So did you manage to get the workaround for it ?
>
> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>
> Regards,
> Vinay Patil
>
> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List
> archive.] <[hidden email]> wrote:
>
> We have the same requirement - we cannot discard any data even if it
> arrives late.
>
>
> - LF
>
>
>
>
> ------------------------------
> *From:* Vinay Patil <[hidden email]>
> *To:* [hidden email]
>
> *Sent:* Sunday, October 2, 2016 8:21 PM
> *Subject:* Regarding Late Elements
>
> Hi Guys,
>
> Just wanted to get an idea on Why Flink decided to completely discard late
> elements in the latest version ?, this was not the case in 1.0.3
>
>
> P.S In our case the data is critical so we cannot discard a single record
> even if it is late, I have written a custom trigger (as suggested by
> Aljoscha) to even accept late elements.
>
>
> Regards,
> Vinay Patil
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user- mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden
> email]
> To unsubscribe from Apache Flink User Mailing List archive., click here.
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
> ------------------------------
> View this message in context: Re: Regarding Late Elements
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com <http://nabble.com>.
>
>
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9307.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9323.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hello LF and Vinay,
With the introduction of “allowed lateness” elements and windows are kept around until the watermark
passes the window.maxTimestamp + allowed_lateness and then they are cleaned up (garbage collected)
Every element that comes in and belongs to a window that is garbage collected is dropped as super-late.
Elements that are late, but no more than the allowed lateness, they are kept the window fires as before.
If you know what the maximum latency is, then the best way is to set the allowed lateness to that value.
Currently Flink drops super-late elements and does not provide any mechanism to manually handle these
elements, BUT there are discussions about adding such a functionality that will allow you to (probably) have
a separate stream with only these elements.
Thanks,
Kostas
> On Oct 3, 2016, at 5:28 PM, lgfmt@yahoo.com wrote:
>
> Not yet.
> I'm hoping a Flink export on this mailing list will reply.
>
>
> - LF
>
>
>
> From: vinay patil <vi...@gmail.com>
> To: user@flink.apache.org
> Sent: Monday, October 3, 2016 8:09 AM
> Subject: Re: Regarding Late Elements
>
> Hi LF,
>
> So did you manage to get the workaround for it ?
>
> I am using a Custom Trigger which is similar to 1.0.3 with few changes
>
> Regards,
> Vinay Patil
>
> On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] <[hidden email] <applewebdata://59CDAE6A-8A95-4A9B-B2E2-23053500938E>> wrote:
> We have the same requirement - we cannot discard any data even if it arrives late.
>
>
> - LF
>
>
>
>
> From: Vinay Patil <[hidden email] <applewebdata://59CDAE6A-8A95-4A9B-B2E2-23053500938E>>
> To: [hidden email] <applewebdata://59CDAE6A-8A95-4A9B-B2E2-23053500938E>
>
> Sent: Sunday, October 2, 2016 8:21 PM
> Subject: Regarding Late Elements
>
> Hi Guys,
>
> Just wanted to get an idea on Why Flink decided to completely discard late elements in the latest version ?, this was not the case in 1.0.3
>
>
> P.S In our case the data is critical so we cannot discard a single record even if it is late, I have written a custom trigger (as suggested by Aljoscha) to even accept late elements.
>
>
> Regards,
> Vinay Patil
>
>
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user- mailing-list-archive.2336050. n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email] <applewebdata://59CDAE6A-8A95-4A9B-B2E2-23053500938E>
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://59CDAE6A-8A95-4A9B-B2E2-23053500938E>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> View this message in context: Re: Regarding Late Elements <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at Nabble.com.
>
>
>
Re: Regarding Late Elements
Posted by lg...@yahoo.com.
Not yet.
I'm hoping a Flink export on this mailing list will reply.
- LF
From: vinay patil <vi...@gmail.com>
To: user@flink.apache.org
Sent: Monday, October 3, 2016 8:09 AM
Subject: Re: Regarding Late Elements
Hi LF,
So did you manage to get the workaround for it ?
I am using a Custom Trigger which is similar to 1.0.3 with few changes
Regards,Vinay Patil
On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:
We have the same requirement - we cannot discard any data even if it arrives late.
- LF
From: Vinay Patil <[hidden email]>
To: [hidden email]
Sent: Sunday, October 2, 2016 8:21 PM
Subject: Regarding Late Elements
Hi Guys,
Just wanted to get an idea on Why Flink decided to completely discard late elements in the latest version ?, this was not the case in 1.0.3
P.S In our case the data is critical so we cannot discard a single record even if it is late, I have written a custom trigger (as suggested by Aljoscha) to even accept late elements.
Regards,Vinay Patil
If you reply to this email, your message will be added to the discussion below: http://apache-flink-user- mailing-list-archive.2336050. n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML
View this message in context: Re: Regarding Late Elements
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by vinay patil <vi...@gmail.com>.
Hi LF,
So did you manage to get the workaround for it ?
I am using a Custom Trigger which is similar to 1.0.3 with few changes
Regards,
Vinay Patil
On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List
archive.] <ml...@n4.nabble.com> wrote:
> We have the same requirement - we cannot discard any data even if it
> arrives late.
>
>
> - LF
>
>
>
>
> ------------------------------
> *From:* Vinay Patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9292&i=0>>
> *To:* [hidden email] <http:///user/SendEmail.jtp?type=node&node=9292&i=1>
> *Sent:* Sunday, October 2, 2016 8:21 PM
> *Subject:* Regarding Late Elements
>
> Hi Guys,
>
> Just wanted to get an idea on Why Flink decided to completely discard late
> elements in the latest version ?, this was not the case in 1.0.3
>
>
> P.S In our case the data is critical so we cannot discard a single record
> even if it is late, I have written a custom trigger (as suggested by
> Aljoscha) to even accept late elements.
>
>
> Regards,
> Vinay Patil
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Regarding-Late-Elements-tp9284p9292.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-Late-Elements-tp9284p9294.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Regarding Late Elements
Posted by lg...@yahoo.com.
We have the same requirement - we cannot discard any data even if it arrives late.
- LF
From: Vinay Patil <vi...@gmail.com>
To: user@flink.apache.org
Sent: Sunday, October 2, 2016 8:21 PM
Subject: Regarding Late Elements
Hi Guys,
Just wanted to get an idea on Why Flink decided to completely discard late elements in the latest version ?, this was not the case in 1.0.3
P.S In our case the data is critical so we cannot discard a single record even if it is late, I have written a custom trigger (as suggested by Aljoscha) to even accept late elements.
Regards,Vinay Patil