You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Urs Schönenberger <ur...@tngtech.com> on 2023/04/23 21:06:34 UTC
Python Datastream: CountTumblingWindowAssigner never purges?
Hi all,
In FLINK-26444, a couple of convenience window assigners were added to
the Python Datastream API, including CountTumblingWindowAssigner. This
assigner uses a CountTrigger by default, which produces TriggerResult.FIRE.
As such, using this window assigner on a data stream will always produce
a "state leak" since older count windows will always be retained without
any chance to work on the elements again.
Unless I am mistaken, the only way to use this assigner would be to
override its trigger with .trigger() afterwards? In this case, should
the default trigger for this assigner not be one with a
TriggerResult.FIRE_AND_PURGE?
Would be happy to file a bug and/or produce a PR for this but wanted to
check with the list first to make sure I'm not missing anything here.
Thanks,
Urs
--
Urs Schönenberger * urs.schoenenberger@tngtech.com
TNG Technology Consulting GmbH, Beta-Str. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Thomas Endres
Aufsichtsratsvorsitzender: Christoph Stock
Sitz: Unterföhring * Amtsgericht München * HRB 135082
Re: Python Datastream: CountTumblingWindowAssigner never purges?
Posted by Dian Fu <di...@gmail.com>.
Filed ticket https://issues.apache.org/jira/browse/FLINK-31949 to track
this issue.
On Thu, Apr 27, 2023 at 11:14 AM Dian Fu <di...@gmail.com> wrote:
> Hi Urs,
>
> I guess you are right. This seems like a bug which should be addressed.
>
> Regards,
> Dian
>
> On Mon, Apr 24, 2023 at 5:07 AM Urs Schönenberger <
> urs.schoenenberger@tngtech.com> wrote:
>
>> Hi all,
>>
>> In FLINK-26444, a couple of convenience window assigners were added to
>> the Python Datastream API, including CountTumblingWindowAssigner. This
>> assigner uses a CountTrigger by default, which produces
>> TriggerResult.FIRE.
>>
>> As such, using this window assigner on a data stream will always produce
>> a "state leak" since older count windows will always be retained without
>> any chance to work on the elements again.
>>
>> Unless I am mistaken, the only way to use this assigner would be to
>> override its trigger with .trigger() afterwards? In this case, should
>> the default trigger for this assigner not be one with a
>> TriggerResult.FIRE_AND_PURGE?
>>
>> Would be happy to file a bug and/or produce a PR for this but wanted to
>> check with the list first to make sure I'm not missing anything here.
>>
>> Thanks,
>> Urs
>>
>> --
>> Urs Schönenberger * urs.schoenenberger@tngtech.com
>> TNG Technology Consulting GmbH, Beta-Str. 13a, 85774 Unterföhring
>> Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Thomas Endres
>> Aufsichtsratsvorsitzender: Christoph Stock
>> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>
>
Re: Python Datastream: CountTumblingWindowAssigner never purges?
Posted by Dian Fu <di...@gmail.com>.
Hi Urs,
I guess you are right. This seems like a bug which should be addressed.
Regards,
Dian
On Mon, Apr 24, 2023 at 5:07 AM Urs Schönenberger <
urs.schoenenberger@tngtech.com> wrote:
> Hi all,
>
> In FLINK-26444, a couple of convenience window assigners were added to
> the Python Datastream API, including CountTumblingWindowAssigner. This
> assigner uses a CountTrigger by default, which produces TriggerResult.FIRE.
>
> As such, using this window assigner on a data stream will always produce
> a "state leak" since older count windows will always be retained without
> any chance to work on the elements again.
>
> Unless I am mistaken, the only way to use this assigner would be to
> override its trigger with .trigger() afterwards? In this case, should
> the default trigger for this assigner not be one with a
> TriggerResult.FIRE_AND_PURGE?
>
> Would be happy to file a bug and/or produce a PR for this but wanted to
> check with the list first to make sure I'm not missing anything here.
>
> Thanks,
> Urs
>
> --
> Urs Schönenberger * urs.schoenenberger@tngtech.com
> TNG Technology Consulting GmbH, Beta-Str. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Thomas Endres
> Aufsichtsratsvorsitzender: Christoph Stock
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>