Re: Kafka Window Store Retention Policy + Outage

["Matthias J. Sax" <mj...@apache.org>]

Sorry for late reply.

> we use a 5 second jumping window with no grace and a 15 minute retention

Why are you using 15 minute retention? Are you using Interactive 
Queries? Because you use no grace, it means that a window would be 
closed right away, and such a larger retention time is not really necessary.


> If my app is brought down for a day or so, which I believe is about the default retention.ms for the window changelog topic when the window retention is short, and the changelog topic is cleaned, am I going to lose windows that have not yet closed and been flushed through the suppress and on to the aggregate?

Well, yes and no... The aggregation operator itself would lose the 
window, however, suppress() that keeps a copy of each window won't. 
(More below.)

Thus, I guess you should be good.


> Is there a reason Kafka specifically adds only a day to the window retention config instead of what feels like the standard/default 7 day retention?

There is no particular reason... In fact, it's changed in 3.0.0 and 
retention time is set according to "window-size + grace-period" if not 
specified otherwise.


> Is there a way to configure a window store to be compacted only, and Kafka Streams can just tombstone the topic when the window is closed/emitted?

No. The problem is, that there is no real "window close event" within 
the aggregation operator. However, suppress() actually works this way. 
That is the reason why suppress() does not lose the window in contrast 
to the aggregation.


> Will a static member that has been down for many hours come back and be assigned the same partitions it had before, and thus re-use its local state?

It depends on your configured `session.timout.ms`. If a static member 
comes back before the session timeout expires, it will get it's old 
assignment. If session timeout hits, a rebalance happens and the 
(static) member is removed from the group.


Exactly-once is orthogonal.


Hope this helps.

-Matthias

On 9/8/21 10:26 AM, Gray, John wrote:
> Hello! I have a somewhat complex question regarding suppressed window stores and their internal topic retention times.
> 
> We have a pretty noisy topic and a pretty expensive aggregate operation, so we window all the input messages into a json array, suppress the output until the window is closed, and then send the array along to our aggregate function. We are using the exactly_once_beta processing.guarantee, setting a group.instance.id to maintain a static group, and using a StatefulSet with persistent storage in Kubernetes. The input topic is noisy so we use a 5 second jumping window with no grace and a 15 minute retention, and a WallClockTimestamp extractor so we don't have to worry about out-of-order events (single topic input) or grace periods.
> 
> My question is: if my app is brought down for a day or so, which I believe is about the default retention.ms for the window changelog topic when the window retention is short, and the changelog topic is cleaned, am I going to lose windows that have not yet closed and been flushed through the suppress and on to the aggregate? If this is true, is a day too small as the default retention.ms? Is there a reason Kafka specifically adds only a day to the window retention config instead of what feels like the standard/default 7 day retention? Is there a way to configure a window store to be compacted only, and Kafka Streams can just tombstone the topic when the window is closed/emitted?
> 
> I would imagine the answer to the above window-loss-question is "yes", unless perhaps the persistent storage and static membership or exactly_once_beta saves me here. Will a static member that has been down for many hours come back and be assigned the same partitions it had before, and thus re-use its local state? Does exactly_once_beta perhaps provide some guarantee to a window being in the suppress changelog (which is compacted) when a thread commits? And perhaps the suppress changelog will hold my to-be-emitted-24-hours-later window?
> 
> Sorry if that is a lot to digest, but thank you for your wisdom!
>