You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephan Pelikan <St...@phactum.at> on 2021/02/02 19:50:44 UTC
Statefun: cancel "sendAfter"
Hi,
I think about using "sendAfter" to implement some kind of timer functionality. I'm wondering if there is no possibility to cancel delayed sent message!
In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?
Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don't want to pollute my state with this because it will inflate the size of my checkpoints.
There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink's docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.
Thanks,
Stephan
AW: Statefun: cancel "sendAfter"
Posted by Stephan Pelikan <St...@phactum.at>.
Hi Gorden,
here is the link, if anyone else is also interested: https://issues.apache.org/jira/browse/FLINK-21308
Cheers,
Stephan
Von: Tzu-Li (Gordon) Tai <tz...@apache.org>
Gesendet: Freitag, 5. Februar 2021 12:58
An: Stephan Pelikan <St...@phactum.at>
Cc: user@flink.apache.org; Igal Shilman <ig...@ververica.com>
Betreff: Re: Statefun: cancel "sendAfter"
Hi Stephan,
Thanks for providing the details of the use case! It does indeed sound like being able to delete scheduled delayed messages would help here.
And yes, please do proceed with creating an issue. As for details on the implementation, we can continue to discuss that on the JIRA.
Cheers,
Gordon
On Wed, Feb 3, 2021 at 3:43 PM Stephan Pelikan <St...@phactum.at>> wrote:
Hi,
thank you Gordon for clarification. My use-case is processing business events of customers. Those events are triggered by ourself or by the customer depending of what’s the current state of the ongoing customer’s business use-case. We need to monitor delayed/missing business events which belong to previous events. For example: the customer has to confirm something we did. Depending on what it is the confirmation has to be within hours, days or even months. If there is a delay we need to know. But if the customer confirms in time we want to cleanup to keep the state small.
I dug a little bit into the code. May I create an issue to discuss my ideas?
Cheers,
Stephan
Von: Tzu-Li (Gordon) Tai <tz...@apache.org>>
Gesendet: Mittwoch, 3. Februar 2021 07:58
An: Stephan Pelikan <St...@phactum.at>>
Cc: user@flink.apache.org<ma...@flink.apache.org>; Igal Shilman <ig...@ververica.com>>
Betreff: Re: Statefun: cancel "sendAfter"
Hi,
You are right, currently StateFun does not support deleting a scheduled delayed message.
StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.
The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.
Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.
Cheers,
Gordon
On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <St...@phactum.at>> wrote:
Hi,
I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!
In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?
Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.
There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.
Thanks,
Stephan
Re: Statefun: cancel "sendAfter"
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Stephan,
Thanks for providing the details of the use case! It does indeed sound like
being able to delete scheduled delayed messages would help here.
And yes, please do proceed with creating an issue. As for details on the
implementation, we can continue to discuss that on the JIRA.
Cheers,
Gordon
On Wed, Feb 3, 2021 at 3:43 PM Stephan Pelikan <St...@phactum.at>
wrote:
> Hi,
>
>
>
> thank you Gordon for clarification. My use-case is processing business
> events of customers. Those events are triggered by ourself or by the
> customer depending of what’s the current state of the ongoing customer’s
> business use-case. We need to monitor delayed/missing business events which
> belong to previous events. For example: the customer has to confirm
> something we did. Depending on what it is the confirmation has to be within
> hours, days or even months. If there is a delay we need to know. But if the
> customer confirms in time we want to cleanup to keep the state small.
>
>
>
> I dug a little bit into the code. May I create an issue to discuss my
> ideas?
>
>
>
> Cheers,
>
> Stephan
>
>
>
>
>
> *Von:* Tzu-Li (Gordon) Tai <tz...@apache.org>
> *Gesendet:* Mittwoch, 3. Februar 2021 07:58
> *An:* Stephan Pelikan <St...@phactum.at>
> *Cc:* user@flink.apache.org; Igal Shilman <ig...@ververica.com>
> *Betreff:* Re: Statefun: cancel "sendAfter"
>
>
>
> Hi,
>
> You are right, currently StateFun does not support deleting a scheduled
> delayed message.
>
> StateFun supports delayed messages by building on top of two Flink
> constructs: 1) registering processing time timers, and 2) buffering the
> message payload to be sent in state.
>
> The delayed messages are kept in the Flink state of the sending operator,
> and timers are registered on the sending operator as well. So technically,
> there doesn't seem to be a blocker for deleting a delayed message and its
> associated timer, if it hasn't been sent yet.
>
> Can you maybe open a JIRA ticket for this, so we have something that
> tracks it?
> Also cc'ing Igal, who might have more comments on whether supporting this
> makes sense.
>
> Cheers,
> Gordon
>
>
>
> On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <St...@phactum.at>
> wrote:
>
> Hi,
>
>
>
> I think about using „sendAfter“ to implement some kind of timer
> functionality. I’m wondering if there is no possibility to cancel delayed
> sent message!
>
>
>
> In my use case it is possible that intermediate events make the delayed
> message obsolete. In some cases the statefun of that certain ID is cleared
> (clear all state variables) and does not exist anymore. In other cases the
> statefun of that ID still exists (and its state). In the latter case I
> could ignore the delayed message, but what about those statefun which do
> not exist anymore?
>
>
>
> Additionally there can be millions of delayed messages which I do not need
> any more and some delays are also hours, days or even months. I don’t want
> to pollute my state with this because it will inflate the size of my
> checkpoints.
>
>
>
> There are no hints in the docs (
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages)
> how those situations are treated. I found in the Flink’s docs that timers
> of keyed processors can be deleted. As far as I know statefuns are based on
> those processors, so I hope that there is something about it. I hope
> someone can clarify what I can expect and how those situations are handled
> internally.
>
>
>
> Thanks,
>
> Stephan
>
>
AW: Statefun: cancel "sendAfter"
Posted by Stephan Pelikan <St...@phactum.at>.
Hi,
thank you Gordon for clarification. My use-case is processing business events of customers. Those events are triggered by ourself or by the customer depending of what’s the current state of the ongoing customer’s business use-case. We need to monitor delayed/missing business events which belong to previous events. For example: the customer has to confirm something we did. Depending on what it is the confirmation has to be within hours, days or even months. If there is a delay we need to know. But if the customer confirms in time we want to cleanup to keep the state small.
I dug a little bit into the code. May I create an issue to discuss my ideas?
Cheers,
Stephan
Von: Tzu-Li (Gordon) Tai <tz...@apache.org>
Gesendet: Mittwoch, 3. Februar 2021 07:58
An: Stephan Pelikan <St...@phactum.at>
Cc: user@flink.apache.org; Igal Shilman <ig...@ververica.com>
Betreff: Re: Statefun: cancel "sendAfter"
Hi,
You are right, currently StateFun does not support deleting a scheduled delayed message.
StateFun supports delayed messages by building on top of two Flink constructs: 1) registering processing time timers, and 2) buffering the message payload to be sent in state.
The delayed messages are kept in the Flink state of the sending operator, and timers are registered on the sending operator as well. So technically, there doesn't seem to be a blocker for deleting a delayed message and its associated timer, if it hasn't been sent yet.
Can you maybe open a JIRA ticket for this, so we have something that tracks it?
Also cc'ing Igal, who might have more comments on whether supporting this makes sense.
Cheers,
Gordon
On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <St...@phactum.at>> wrote:
Hi,
I think about using „sendAfter“ to implement some kind of timer functionality. I’m wondering if there is no possibility to cancel delayed sent message!
In my use case it is possible that intermediate events make the delayed message obsolete. In some cases the statefun of that certain ID is cleared (clear all state variables) and does not exist anymore. In other cases the statefun of that ID still exists (and its state). In the latter case I could ignore the delayed message, but what about those statefun which do not exist anymore?
Additionally there can be millions of delayed messages which I do not need any more and some delays are also hours, days or even months. I don’t want to pollute my state with this because it will inflate the size of my checkpoints.
There are no hints in the docs (https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages) how those situations are treated. I found in the Flink’s docs that timers of keyed processors can be deleted. As far as I know statefuns are based on those processors, so I hope that there is something about it. I hope someone can clarify what I can expect and how those situations are handled internally.
Thanks,
Stephan
Re: Statefun: cancel "sendAfter"
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
You are right, currently StateFun does not support deleting a scheduled
delayed message.
StateFun supports delayed messages by building on top of two Flink
constructs: 1) registering processing time timers, and 2) buffering the
message payload to be sent in state.
The delayed messages are kept in the Flink state of the sending operator,
and timers are registered on the sending operator as well. So technically,
there doesn't seem to be a blocker for deleting a delayed message and its
associated timer, if it hasn't been sent yet.
Can you maybe open a JIRA ticket for this, so we have something that tracks
it?
Also cc'ing Igal, who might have more comments on whether supporting this
makes sense.
Cheers,
Gordon
On Wed, Feb 3, 2021 at 3:51 AM Stephan Pelikan <St...@phactum.at>
wrote:
> Hi,
>
>
>
> I think about using „sendAfter“ to implement some kind of timer
> functionality. I’m wondering if there is no possibility to cancel delayed
> sent message!
>
>
>
> In my use case it is possible that intermediate events make the delayed
> message obsolete. In some cases the statefun of that certain ID is cleared
> (clear all state variables) and does not exist anymore. In other cases the
> statefun of that ID still exists (and its state). In the latter case I
> could ignore the delayed message, but what about those statefun which do
> not exist anymore?
>
>
>
> Additionally there can be millions of delayed messages which I do not need
> any more and some delays are also hours, days or even months. I don’t want
> to pollute my state with this because it will inflate the size of my
> checkpoints.
>
>
>
> There are no hints in the docs (
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/java.html#sending-delayed-messages)
> how those situations are treated. I found in the Flink’s docs that timers
> of keyed processors can be deleted. As far as I know statefuns are based on
> those processors, so I hope that there is something about it. I hope
> someone can clarify what I can expect and how those situations are handled
> internally.
>
>
>
> Thanks,
>
> Stephan
>