Dynamin Windowing in with Pyflink

[Nawaz Nayeem via user <us...@flink.apache.org>]

Hey, I’ve been trying to emulate the behavior of a dynamic window, as Flink
does not support dynamic window sizes. My operator inherits from
KeyedProcessFunction, and I’m only using KeyedStates to manipulate the
window_size. I’m clearing the KeyedStates when my bucket(window) is
complete, to reset the bucket size.

My concern is, as Flink does not support dynamic windows, is this approach
going against Flink Architecture? Like will it break checkpointing
mechanism in distributed systems? It's been noted that I’m only using
KeyedStates for maintaining or implementing the dynamic window.


Any feedback would be appreciated.
Thank you.

-- 









SELISE Group
Zürich: The Circle 37, 8058 Zürich-Airport, 
Switzerland
Munich: Tal 44, 80331 München, Germany
Dubai: Building 3, 3rd 
Floor, Dubai Design District, Dubai, United Arab Emirates
Dhaka: Midas 
Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh
Thimphu: Bhutan 
Innovation Tech Center, Babesa, P.O. Box 633, Thimphu, Bhutan

Visit us: 
www.selisegroup.com <http://www.selisegroup.com>




-- 




*Important Note: This e-mail and any attachment are confidential and 
may contain trade secrets and may well also be legally privileged or 
otherwise protected from disclosure. If you have received it in error, you 
are on notice of its status. Please notify us immediately by reply e-mail 
and then delete this e-mail and any attachment from your system. If you are 
not the intended recipient please understand that you must not copy this 
e-mail or any attachment or disclose the contents to any other person. 
Thank you for your cooperation.*

Re: Dynamin Windowing in with Pyflink

[Dian Fu <di...@gmail.com>]

Hi Nawaz,

>> My concern is, as Flink does not support dynamic windows, is this
approach going against Flink Architecture.
Per my understanding, the session window could be seen as a kind of dynamic
window. Besides, Flink also supports user-defined window with which users
should also be able to implement dynamic window. So I think what you are
doing is a good solution and will not be against Flink Architecture.

Regards,
Dian

On Tue, May 16, 2023 at 7:08 PM Nawaz Nayeem via user <us...@flink.apache.org>
wrote:

> Hey, I’ve been trying to emulate the behavior of a dynamic window, as
> Flink does not support dynamic window sizes. My operator inherits from
> KeyedProcessFunction, and I’m only using KeyedStates to manipulate the
> window_size. I’m clearing the KeyedStates when my bucket(window) is
> complete, to reset the bucket size.
>
> My concern is, as Flink does not support dynamic windows, is this approach
> going against Flink Architecture? Like will it break checkpointing
> mechanism in distributed systems? It's been noted that I’m only using
> KeyedStates for maintaining or implementing the dynamic window.
>
>
> Any feedback would be appreciated.
> Thank you.
>
> [image: SELISE]
>
> SELISE Group
> Zürich: The Circle 37, 8058 Zürich-Airport, Switzerland
> Munich: Tal 44, 80331 München, Germany
> Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab
> Emirates
> Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh
> Thimphu: Bhutan Innovation Tech Center, Babesa, P.O. Box 633, Thimphu,
> Bhutan
>
> Visit us: www.selisegroup.com
>
> *Important Note: This e-mail and any attachment are confidential and may
> contain trade secrets and may well also be legally privileged or otherwise
> protected from disclosure. If you have received it in error, you are on
> notice of its status. Please notify us immediately by reply e-mail and then
> delete this e-mail and any attachment from your system. If you are not the
> intended recipient please understand that you must not copy this e-mail or
> any attachment or disclose the contents to any other person. Thank you for
> your cooperation.*
>