clear State using business logic

[Sigalit Eliazov <e....@gmail.com>]

Hi all,

the flow in our pipeline is:

1. read event X from kafka. open fixed window of 30 sec.
2. read event subscription from kafka. open GlobalWindow and store a
state of all subscriptions.
3. match X and Y using key and if there is a match send an event to
another kafka topic. (we use the state as side input)

if a user unsubscribes (meaning we read from a different source an
event of unsubscribe)  we would like to delete the relevant entry from
the state.
Can this be achieved only using state and not using some external cache/db?

I am aware there is an option to add timers on state but the
expiration logic is not time based.
Any suggestions?

Thanks in advance
Sigalit

Re: clear State using business logic

[Reza Ardeshir Rokni <ra...@gmail.com>]

Have you explored processing time timers?

https://beam.apache.org/releases/javadoc/2.43.0/org/apache/beam/sdk/state/TimeDomain.html#PROCESSING_TIME

On Wed, 23 Nov 2022 at 13:46, Sigalit Eliazov <e....@gmail.com> wrote:

> Hi all,
>
> the flow in our pipeline is:
>
> 1. read event X from kafka. open fixed window of 30 sec.
> 2. read event subscription from kafka. open GlobalWindow and store a
> state of all subscriptions.
> 3. match X and Y using key and if there is a match send an event to
> another kafka topic. (we use the state as side input)
>
> if a user unsubscribes (meaning we read from a different source an
> event of unsubscribe)  we would like to delete the relevant entry from
> the state.
> Can this be achieved only using state and not using some external cache/db?
>
> I am aware there is an option to add timers on state but the
> expiration logic is not time based.
> Any suggestions?
>
> Thanks in advance
> Sigalit
>