[GitHub] [kafka] cadonna commented on pull request #10953: MINOR: Default GRACE with Old API should set as 24H minus window-size / inactivity-gap

[GitBox <gi...@apache.org>]

cadonna commented on pull request #10953:
URL: https://github.com/apache/kafka/pull/10953#issuecomment-872811076


   Some details for the other reviewers.
   
   In 2.8 and before, we computed the default grace period with `Math.max(maintainDurationMs - sizeMs, 0);` in method `gracePeriodMs()` in `TimeWindows`, `SessionWindows`, and `JoinWindows`. That means that the default grace period has never been 24 hours but 24 hours - window size. Since `gracePeriodMs()` is used to compute the retention time of the changelog topic for the corresponding state store it is important to keep the same computation for the deprecated methods. Otherwise, Streams app that run with 2.8 and before might not be compatible with Streams 3.0 because the retention time of the changelog topics created with older Streams apps will be smaller than the assumed retention time for Streams apps in 3.0. For example, with a window size of 10 hours, an old Streams app would have created a changelog topic with retention time 10 hours (window size) + 14 hours (default grace period, 24 hours - 10 hours). A 3.0 Streams app would assume a retention time of 10 hours (window size)
  + 24 hours (deprecated default grace period as currently specified on trunk). In the presence of failures, where a state store needs to recreated, records might get lost, because before the failure the state store of a 3.0 Streams app contained 10 hours + 24 hours of records whereas the changelog topic that was created with the old Streams app would only contain 10 hours + 14 hours of records.
   
   All this happened due to us always stating that the default grace period was 24 hours although it was not completely correct and a connected and unfortunate misunderstanding when we removed deprecated windows APIs (#10378).      
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org