[jira] [Commented] (FLINK-5572) ListState in SlidingEventTimeWindow and SlidingProcessingTimeWindow optimization

["Syinchwun Leo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853453#comment-15853453 ] 

Syinchwun Leo commented on FLINK-5572:
--------------------------------------

I try to find a common way to abstract the States in WindowOperator, and it is very hard to find a balance between memory and performance. The current framework is a good method to express most States except ListState. ListState is commonly used in UDF Windows. ListState keeps raw data, and other States may keep intermediate data. It's hard to unite all States using a common way to keep, fire and clean up window. If we can keep the current framework, and only optimize each specific State?

> ListState in SlidingEventTimeWindow and SlidingProcessingTimeWindow optimization
> --------------------------------------------------------------------------------
>
>                 Key: FLINK-5572
>                 URL: https://issues.apache.org/jira/browse/FLINK-5572
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API
>    Affects Versions: 1.2.1
>         Environment: CentOS 7.2
>            Reporter: Syinchwun Leo
>
> When using ListState in SlidingEventTimeWindow and SlidingProcessingTimeWindow, an element  may be assigned to multiple overlapped windows. It may lead to storage consuming. for example, 
> window(SlidingEventTimeWindows.of(Time.seconds(10), Time.seconds(2))).apply(UDF window function), each element is assigned to 5 windows, When the window size is very large, it is unacceptable(size/slide is very large).
> We plan to make a little optimization, and the doc is in 
> https://docs.google.com/document/d/1HCt1Si3YNGFwsl2H5SO0f7WD69DdBBPVJA6abd3oFWo/edit?usp=sharing 
> Comments?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)