[jira] [Commented] (KAFKA-9524) Default window retention does not consider grace period

["John Roesler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032714#comment-17032714 ] 

John Roesler commented on KAFKA-9524:
-------------------------------------

Thanks for the report, [~mikebin]!

FWIW, the workaround is to also set the retention time on the store (using Materialized) to be at least windowSize + grace. I just mention that for anyone who happens to find this ticket after searching the error message.

I agree that we could and should configure the store's retention properly. If I recall correctly, we need to remove the deprecated method:
> org.apache.kafka.streams.kstream.TimeWindows#maintainMs
and the other deprecated members of Windows and its children.

They've been deprecated since 2.1, so it's probably been long enough by now for us to go ahead and remove them and make Streams behave more nicely in this case.

> Default window retention does not consider grace period
> -------------------------------------------------------
>
>                 Key: KAFKA-9524
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9524
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 2.4.0
>            Reporter: Michael Bingham
>            Priority: Minor
>
> In a windowed aggregation, if you specify a window size larger than the default window retention (1 day), Streams will implicitly set retention accordingly to accommodate windows of that size. For example, 
> {code:java}
> .windowedBy(TimeWindows.of(Duration.ofDays(20))) 
> {code}
> In this case, Streams will implicitly set window retention to 20 days, and no exceptions will occur.
> However, if you also include a non-zero grace period on the window, such as:
> {code:java}
> .windowedBy(TimeWindows.of(Duration.ofDays(20)).grace(Duration.ofMinutes(5)))
> {code}
> In this case, Streams will still implicitly set the window retention 20 days (not 20 days + 5 minutes grace), and an exception will be thrown:
> {code:java}
> Exception in thread "main" java.lang.IllegalArgumentException: The retention period of the window store KSTREAM-KEY-SELECT-0000000002 must be no smaller than its window size plus the grace period. Got size=[1728000000], grace=[300000], retention=[1728000000]{code}
> Ideally, Streams should include grace period when implicitly setting window retention.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)