[jira] [Commented] (FLINK-5387) Lightweight Event Time Window

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802246#comment-15802246 ] 

Stephan Ewen commented on FLINK-5387:
-------------------------------------

There have been a few discussions about aligned windows and pane optimization. Have a look here at this variant of the aligned window operators:

https://github.com/apache/flink/blob/release-1.1.4/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/AccumulatingProcessingTimeWindowOperator.java
https://github.com/apache/flink/blob/release-1.1.4/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/AbstractAlignedProcessingTimeWindowOperator.java

They are used automatically in Flink 1.1 for cases of simple sliding windows on processing time.
You can use them in Flink 1.2 with special window assigners.

In the end it turns out that this approach is actually not always superior in performance - it depends very much on the density of events versus window pane size.


> Lightweight Event Time Window
> -----------------------------
>
>                 Key: FLINK-5387
>                 URL: https://issues.apache.org/jira/browse/FLINK-5387
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API
>    Affects Versions: 1.1.3
>            Reporter: Syinchwun Leo
>
> When a new data coming and insert into Event Time Window, windowSize/slide subWindows are generated. and the data is copied to all of these subWindows. When the value of windowSize/slide is very big, redundancy is large, and this will slow the RocksDB's checkpoint. Event Window should redesigned, and candidate design is similar as Session window, only generating a pane(aligned with slide) when a new data come instead of generating a series of subWindows 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)