[jira] [Commented] (FLINK-3714) Add Support for "Allowed Lateness"

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326055#comment-15326055 ] 

ASF GitHub Bot commented on FLINK-3714:
---------------------------------------

GitHub user kl0u opened a pull request:

    https://github.com/apache/flink/pull/2093

    [FLINK-3714] Add Support for "Allowed Lateness"

    Allows the user to specify an "allowed lateness" for  elements, before they are dropped as late arrivals.
    In addition, it defines a "clean up" time for the window state, which is the end of the window plus the 
    aforementioned allowed lateness. When the watermark or the processing time (depending on the widow
    assigner) passes this mark, then the window state is purged/cleaned.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kl0u/flink lateness_n_gc

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2093.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2093
    
----
commit be294b6dde8c830bde6d6944c403187d366f1c80
Author: kl0u <kk...@gmail.com>
Date:   2016-05-31T15:13:58Z

    [FLINK-3714] Add Support for "Allowed Lateness"
    
    Handle late elements and take care
    of cleaning the window state.

----


> Add Support for "Allowed Lateness"
> ----------------------------------
>
>                 Key: FLINK-3714
>                 URL: https://issues.apache.org/jira/browse/FLINK-3714
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Streaming
>            Reporter: Aljoscha Krettek
>            Assignee: Kostas Kloudas
>
> As mentioned in https://docs.google.com/document/d/1Xp-YBf87vLTduYSivgqWVEMjYUmkA-hyb4muX3KRl08/edit# we should add support for an allowed lateness setting.
> This includes several things:
>  - API for setting allowed lateness
>  - Dropping of late elements 
>  - Garbage collection of windows state/timers
> Depending on whether the {{WindowAssigner}} assigns windows based on event time or processing time we have to adjust the GC behavior. For event-time windows "allowed lateness" makes sense and we should garbage collect after this expires. For processing-time windows "allowed lateness" does not make sense and we should always GC window state/timers at the end timestamp of a processing-time window. I think that we need a method for this on {{WindowAssigner}} that allows to differentiate between event-time windows and processing-time windows: {{boolean WindowAssigner.isEventTime()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)