[jira] [Commented] (KAFKA-14172) bug: State stores lose state when tasks are reassigned under EOS wit…

["John Gray (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645330#comment-17645330 ] 

John Gray commented on KAFKA-14172:
-----------------------------------

I do hope this bug gets some eyes eventually, it has been pretty devastating for us. I amend my previous statement that this is related to our brokers rolling, it actually seems it can happen any time our restore consumers in EOS apps w/ big state try to restore said big state. Sadly, setting the acceptable.lag config does not help us because we do not have the extra space to run standby threads/replicas. 

We actually had to resort to dumping our keys to a sqlserver and then querying for "should this key exist?" when we are pulling keys from our state store. If the state store returns nothing and sqlserver says it has seen that key before, we kill the app, which then causes us to pull the state again, which usually fixes the issue. Something is _very_ wrong with these restore consumers (or I am doing something horribly wrong in this app, although we never had this problem before Kafka 3.0.0 or 3.1.0).

> bug: State stores lose state when tasks are reassigned under EOS wit…
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-14172
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14172
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 3.1.1
>            Reporter: Martin Hørslev
>            Priority: Major
>
> h1. State stores lose state when tasks are reassigned under EOS with standby replicas and default acceptable lag.
> I have observed that state stores used in a transform step under a Exactly Once semantics ends up losing state after a rebalancing event that includes reassignment of tasks to previous standby task within the acceptable standby lag.
>  
> The problem is reproduceable and an integration test have been created to showcase the [issue|https://github.com/apache/kafka/pull/12540]. 
> A detailed description of the observed issue is provided [here|https://github.com/apache/kafka/pull/12540/files?short_path=3ca480e#diff-3ca480ef093a1faa18912e1ebc679be492b341147b96d7a85bda59911228ef45]
> Similar issues have been observed and reported to StackOverflow for example [here|https://stackoverflow.com/questions/69038181/kafka-streams-aggregation-data-loss-between-instance-restarts-and-rebalances].
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)