Sliding Window - Weird behaviour

[madhairsilence <ha...@tcs.com>]

I have a datastream
1,2,3,4,5,6,7....

I applied a sliding countWindow as
inputStream.keyBy("num").countWindow(2,1)

I expect an output as
1,2
2,3
3,4

But am getting an output as
1
1,2
2,3
3,4

Why does the data slide first and then accumulate the window size



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Sliding-Window-Weird-behaviour-tp17013.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

RE: Sliding Window - Weird behaviour

[Radu Tudoran <ra...@huawei.com>]

Hi,

You need to implement your own timer. You do this when you create your window by assigning the timer. In your custom timer you would need to implement the desired logic in the onElement method.
You can keep a counter that you increment for each element up to your desired number of elements and FIRE only when this value is reaches your threshold after which you want to trigger

You can take a look in existing triggers
https://github.com/apache/flink/tree/1875cac03042dad4a4c47b0de8364f02fbe457c6/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers 


-----Original Message-----
From: madhairsilence [mailto:harish.kumard@tcs.com] 
Sent: Thursday, April 13, 2017 12:25 PM
To: dev@flink.apache.org
Subject: Re: Sliding Window - Weird behaviour

Hi Xingcan

Thanks for the answer. But up to my understanding

countWindow(4,2) - Should wait for 4 elements (or window not more than 4
element) and once the window is ready, slide two items

Now if I have to stopped asking why questions and worry about my current problem, how do I achieve this expected output.

Stream : 1,2,3,4,5,6,7,8...

Output:
1,2
2,3
3,4
4,5...



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Sliding-Window-Weird-behaviour-tp17013p17019.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: Sliding Window - Weird behaviour

[madhairsilence <ha...@tcs.com>]

Hi Xingcan

Thanks for the answer. But up to my understanding

countWindow(4,2) - Should wait for 4 elements (or window not more than 4
element) and once the window is ready, slide two items

Now if I have to stopped asking why questions and worry about my current
problem, how do I achieve this expected output.

Stream : 1,2,3,4,5,6,7,8...

Output:
1,2
2,3
3,4
4,5...



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Sliding-Window-Weird-behaviour-tp17013p17019.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Re: Sliding Window - Weird behaviour

[Xingcan Cui <xi...@gmail.com>]

​Hi harish,

I will not argue for the correctness of the result​s, but just tell you why
this happens.

The countWindow(2, 1) can be regarded as two separate processes: 1)
maintain a window whose size *not exceeds* 2 and 2) trigger window
evaluation every single record.

Actually, in Flink the two processes execute independently and that's why
the first record 1 triggered window accumulation in you example.

Hope this helps,
Xingcan

On Thu, Apr 13, 2017 at 4:43 PM, madhairsilence <ha...@tcs.com>
wrote:

> I have a datastream
> 1,2,3,4,5,6,7....
>
> I applied a sliding countWindow as
> inputStream.keyBy("num").countWindow(2,1)
>
> I expect an output as
> 1,2
> 2,3
> 3,4
>
> But am getting an output as
> 1
> 1,2
> 2,3
> 3,4
>
> Why does the data slide first and then accumulate the window size
>
>
>
> --
> View this message in context: http://apache-flink-mailing-
> list-archive.1008284.n3.nabble.com/Sliding-Window-
> Weird-behaviour-tp17013.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>