[jira] [Commented] (KAFKA-19853) Transaction Failure when StreamThread blocks on StateUpdater during onAssignment()

["Colt McNealy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093073#comment-18093073 ] 

Colt McNealy commented on KAFKA-19853:
--------------------------------------

[~lucasbru] yes, we can continue to reproduce this occasionally with the following sequence of events.
 # Build a cluster with 200GB state per instance, 3 instances (600GB total), 1 standby replica, 4 CPU's, 3 stream threads
 # Kill one instance completely.
 # The cluster has no offline tasks since standby's recover, but both remaining instances are now full-speed-ahead restoring the missing standby's.
 # Trigger a rebalance among the last standing servers, normally by restarting one of them.

We can reduce the likelihood of the stall issue observed above happening by:
 # Increasing the available CPU's
 # Increasing disk bandwidth
 # Reducing the number of stream threads (so there are fewer state updaters thrashing the rocksdb compaction threadpool and disk bandwidth)

I believe this bug is still valid; also, our RocksDB configurations aren't that crazy from normal. We mostly follow the Confluent guide for limiting memory, and set it to use about 60% of container memory. 25% is used by the JVM heap, and the remaining 15% is buffer overhead.

> Transaction Failure when StreamThread blocks on StateUpdater during onAssignment()
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-19853
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19853
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 3.9.0
>            Reporter: Colt McNealy
>            Assignee: Colt McNealy
>            Priority: Major
>         Attachments: image (3).png, image (4).png, image (5).png
>
>
> We've observed that the `StreamThread` blocks waiting for a `Future` from the `StateUpdater` in the `StreamsPartitionAssigner#onAssignment()` method when we are moving a task out of the `StateUpdater` and onto the `StreamThread`.
>  
> This can cause problems because, during restoration or with warmup replicas, the `StateUpdater#runOnce()` method can take a long time (upwards of 20 seconds) when RocksDB stalls writes to allow compaction to keep up. In EOS this blockage may cause the transaction to time out, which is a big mess. This is because the `StreamThread` may have an open transaction before the `StreamsPartitionAssignor#onAssignment()` method is called.
>  
> Some screenshots from the JFR below (credit to [~eduwerc]).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)