[jira] [Comment Edited] (KAFKA-20416) RocksDB loses entries during broker patches.

["Bill Bejeck (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110815#comment-18110815 ] 

Bill Bejeck edited comment on KAFKA-20416 at 9/15/26 2:40 PM:
--------------------------------------------------------------

[~fabianbell] Here's the PR -[https://github.com/apache/kafka/pull/23339]-

 

After taking a deeper look, I'm thinking we are not going ahead with the PR for Kafka Streams. The double-race we identified (Race A: coordinator resignation returns a transient committedOffset = -1 / Errors.NONE for a group that has offsets → client coerces to 0; Race B: concurrent EOS TaskCorruptedException wipes the store → the 0 restore ceiling declares the wiped source-as-changelog store fully restored while empty) is only reachable with the old group coordinator, and only through the narrow resignation/unload window where handleFetchOffsets does its status check and getOffsets read under two separate lock acquisitions. Against the new (KIP-848) coordinator the fetch and unload run under the same context.lock and are ACTIVE-gated, so there is no check-and-get gap — Race A is not reachable.  Given the old group coordinator has been removed this isn't an issue.


was (Author: bbejeck):
[~fabianbell] Here's the PR https://github.com/apache/kafka/pull/23339

> RocksDB loses entries during broker patches.
> --------------------------------------------
>
>                 Key: KAFKA-20416
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20416
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 3.9.1
>         Environment: Broker: MSK with kafka.m7g.2xlarge instances
> Application: EKS with m7i.2xlarge instances and Bottlerocket OS 1.57.0
>            Reporter: Fabian Bell
>            Assignee: Bill Bejeck
>            Priority: Major
>         Attachments: logs
>
>
> h2. Problem:
> We discovered a strange behaviour on our production environment. We use a KTable to look up data from a topic we write to.
>  
> {code:java}
> builder.table(topicName, Consumed.with(keySerde, valueSerde), Materialized.as(storeName)) {code}
>  
> When we access the store in the processor, we observed that the store returned null values for keys that have non-null entries in the topic that backs the KTable after an MSK security patch. We never tombstone an entry in our topic nor have a delete retention activated.
> This only happens for some of our instances.
> We see the following stream logs:
>  
> {code:java}
> Committing task(s) 0_14 failed.
> Detected the states of tasks [0_14] are corrupted. Will close the task as dirty and re-create and bootstrap from scratch.
> Active task(s) got corrupted. Triggering a rebalance.
> End offset for changelog our-topic-14 initialized as 16596290.
> Restoration in progress for 1 partitions. {our-topic-14: position=0, end=16596290, totalRestored=0}
> State transition from RUNNING to PARTITIONS_REVOKED
> No followup rebalance was requested, resetting the rebalance schedule.
> partition revocation took 80 ms.
> State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> State transition from PARTITIONS_ASSIGNED to RUNNING {code}
>  
> This all happens within a few seconds, and the `Restoration in progress ...` log is the only one we can see. A full restoration usually takes like 30 min. 
> The error message of the commit failure is 
> {code:java}
> o.a.k.c.e.TimeoutException: Timeout expired after 60000ms while awaiting AddOffsetsToTxn {code}
> We can fix this situation by clearing the state directory and forcing a full restoration.
> h2. Context:
> Each instance has its own persistent state directory. The configured state directory does  not change.  
> Processing Guarantee: exactly_once_v2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)