You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Matthias J. Sax (Jira)" <ji...@apache.org> on 2021/09/10 04:57:00 UTC
[jira] [Assigned] (KAFKA-13216) Streams left/outer joins cause new
internal changelog topic to grow unbounded
[ https://issues.apache.org/jira/browse/KAFKA-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias J. Sax reassigned KAFKA-13216:
---------------------------------------
Assignee: Guozhang Wang
> Streams left/outer joins cause new internal changelog topic to grow unbounded
> -----------------------------------------------------------------------------
>
> Key: KAFKA-13216
> URL: https://issues.apache.org/jira/browse/KAFKA-13216
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Sergio Peña
> Assignee: Guozhang Wang
> Priority: Critical
> Fix For: 3.1.0
>
>
> This bug is caused by the improvements made in https://issues.apache.org/jira/browse/KAFKA-10847, which fixes an issue with stream-stream left/outer joins. The issue is only caused when a stream-stream left/outer join is used with the new `JoinWindows.ofTimeDifferenceAndGrace()` API that specifies the window time + grace period. This new API was added in AK 3.0. No previous users are affected.
> The issue causes that the internal changelog topic used by the new OUTERSHARED window store keeps growing unbounded as new records come. The topic is never cleaned up nor compacted even if tombstones are written to delete the joined and/or expired records from the window store. The problem is caused by a parameter required in the window store to retain duplicates. This config causes that tombstones records have a new sequence ID as part of the key ID in the changelog making those keys unique. Thus causing the cleanup policy not working.
> In 3.0, we deprecated {{JoinWindows.of(size)}} in favor of {{JoinWindows.ofTimeDifferenceAndGrace()}} -- the old API uses the old semantics and is thus not affected while the new API enable the new semantics; the problem is that we deprecated the old API and thus tell users that they should switch to the new broken API.
> We have two ways forward:
> * Fix the bug (non trivial)
> * Un-deprecate the old {{JoinWindow.of(size)}} API (and tell users not to use the new but broken API)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)