You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Kumar Bolar, Harshith" <hk...@arity.com> on 2019/02/14 10:53:17 UTC
How to clear state immediately after a keyed window is processed?
Hi all,
My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.
Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?
Thanks,
Harshith
Re: Re: Re: How to clear state immediately after a keyed window is
processed?
Posted by "Kumar Bolar, Harshith" <hk...@arity.com>.
Thanks a lot :D
From: Konstantin Knauf <ko...@ververica.com>
Date: Thursday, 14 February 2019 at 5:38 PM
To: Harshith Kumar Bolar <hk...@arity.com>, user <us...@flink.apache.org>
Subject: [External] Re: Re: How to clear state immediately after a keyed window is processed?
Yes, for processing-time windows the clean up time is exactly the end time of the window, because by definition there is no late data and state does not need to be kept around.
On Thu, Feb 14, 2019 at 1:03 PM Kumar Bolar, Harshith <hk...@arity.com>> wrote:
Thanks Konstanin,
But I’m using processing time, hence no watermarks. Will the state still be cleared automatically if nothing is done?
From: Konstantin Knauf <ko...@ververica.com>>
Date: Thursday, 14 February 2019 at 5:18 PM
To: Harshith Kumar Bolar <hk...@arity.com>>
Cc: "user@flink.apache.org<ma...@flink.apache.org>" <us...@flink.apache.org>>
Subject: [External] Re: How to clear state immediately after a keyed window is processed?
Hi Harshith,
when you use Flink's Windowing API, the state of an event time window is cleared once the watermark passes the end time of the window (that's when the window fires) + the allowed lateness. So, as long as you don't configure additional allowed lateness (default=0), Flink will already behave as described by default.
Cheers,
Konstantin
On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <hk...@arity.com>> wrote:
Hi all,
My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.
Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?
Thanks,
Harshith
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
Error! Filename not specified.<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ververica.com_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=N-AULj-Zgx6gLH2HLY3aoBLwWtlE4Htr9RAdZnfNFLU&s=xPlFRgXSEpjXe1cD_CCotZiqdo6DvuAqCtNP3fZUK6Q&e=>
Follow us @VervericaData
--
Join Flink Forward<https://urldefense.proofpoint.com/v2/url?u=https-3A__flink-2Dforward.org_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=N-AULj-Zgx6gLH2HLY3aoBLwWtlE4Htr9RAdZnfNFLU&s=0euY5VWL9xIEpeJNY89ujxewLoPeALdyGVGiUpYbGxQ&e=> - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
[mage removed by sender.]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ververica.com_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=ZDZ8Y523QTmoeyfaljSQ3m3mZd4j9s8beYn5wgFDfqI&s=nWLC2eIp4VWMuCGRd3dhVwy2o-YHyDc0Lrf8zSeUjGw&e=>
Follow us @VervericaData
--
Join Flink Forward<https://urldefense.proofpoint.com/v2/url?u=https-3A__flink-2Dforward.org_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=ZDZ8Y523QTmoeyfaljSQ3m3mZd4j9s8beYn5wgFDfqI&s=5ooTBV09vDx4ODAiNSjngPZjNw6OSzI9H9Spwdn5UCo&e=> - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Re: How to clear state immediately after a keyed window is processed?
Posted by Konstantin Knauf <ko...@ververica.com>.
Yes, for processing-time windows the clean up time is exactly the end time
of the window, because by definition there is no late data and state does
not need to be kept around.
On Thu, Feb 14, 2019 at 1:03 PM Kumar Bolar, Harshith <hk...@arity.com>
wrote:
> Thanks Konstanin,
>
>
>
> But I’m using processing time, hence no watermarks. Will the state still
> be cleared automatically if nothing is done?
>
>
>
> *From: *Konstantin Knauf <ko...@ververica.com>
> *Date: *Thursday, 14 February 2019 at 5:18 PM
> *To: *Harshith Kumar Bolar <hk...@arity.com>
> *Cc: *"user@flink.apache.org" <us...@flink.apache.org>
> *Subject: *[External] Re: How to clear state immediately after a keyed
> window is processed?
>
>
>
> Hi Harshith,
>
>
>
> when you use Flink's Windowing API, the state of an event time window is
> cleared once the watermark passes the end time of the window (that's when
> the window fires) + the allowed lateness. So, as long as you don't
> configure additional allowed lateness (default=0), Flink will already
> behave as described by default.
>
>
>
> Cheers,
>
>
>
> Konstantin
>
>
>
> On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <hk...@arity.com>
> wrote:
>
> Hi all,
>
>
>
> My application uses a keyed window that is keyed by a function of
> timestamp. This means once that particular window has been fired and
> processed, there is no use in keeping that key active because there is no
> way that particular key will appear again. Because this use case involves
> continuously expanding keys, I want to clear the state of a key as soon as
> it is finished processing without having to configure timers.
>
>
>
> Is this something that can be achieved in the evictor method
> or apply method after each keyed window is done processing?
>
>
>
> Thanks,
>
> Harshith
>
>
>
>
>
>
>
> --
>
> *Konstantin Knauf* | Solutions Architect
>
> +49 160 91394525
>
>
>
> [image: mage removed by sender.]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ververica.com_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=N-AULj-Zgx6gLH2HLY3aoBLwWtlE4Htr9RAdZnfNFLU&s=xPlFRgXSEpjXe1cD_CCotZiqdo6DvuAqCtNP3fZUK6Q&e=>
>
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__flink-2Dforward.org_&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=N-AULj-Zgx6gLH2HLY3aoBLwWtlE4Htr9RAdZnfNFLU&s=0euY5VWL9xIEpeJNY89ujxewLoPeALdyGVGiUpYbGxQ&e=>
> - The Apache Flink Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: How to clear state immediately after a keyed window is processed?
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Harshith,
when you use Flink's Windowing API, the state of an event time window is
cleared once the watermark passes the end time of the window (that's when
the window fires) + the allowed lateness. So, as long as you don't
configure additional allowed lateness (default=0), Flink will already
behave as described by default.
Cheers,
Konstantin
On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <hk...@arity.com>
wrote:
> Hi all,
>
>
>
> My application uses a keyed window that is keyed by a function of
> timestamp. This means once that particular window has been fired and
> processed, there is no use in keeping that key active because there is no
> way that particular key will appear again. Because this use case involves
> continuously expanding keys, I want to clear the state of a key as soon as
> it is finished processing without having to configure timers.
>
>
>
> Is this something that can be achieved in the evictor method
> or apply method after each keyed window is done processing?
>
>
>
> Thanks,
>
> Harshith
>
>
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen