[jira] [Created] (KAFKA-21356) Performance issue in share fetch with share group DLQ enabled.

["Sushant Mahajan (Jira)" <ji...@apache.org>]

Sushant Mahajan created KAFKA-21356:
---------------------------------------

             Summary: Performance issue in share fetch with share group DLQ enabled.
                 Key: KAFKA-21356
                 URL: https://issues.apache.org/jira/browse/KAFKA-21356
             Project: Kafka
          Issue Type: Bug
            Reporter: Sushant Mahajan
            Assignee: Sushant Mahajan


Enabling DLQ on a share group record could strangulate the overall share fetch throughput by choking the persister. The causes are:
1. `ShareCoordinatorMetadataCacheHelperImpl.isDlqEnabledOnTopic` returns the value of a dynamic config so it is called repeatedly. The impl of this method creates a new LogConfig object, on each invocation and parses entire TopicConfig. On a system with large number of DLQ requests, these calls add up consuming CPU and choking the persister.
Fix: Use raw config lookup (shrinks cpu samples at high load from 27% to 0.1%)

2. Share group DLQ requires reading data from the ReplicaManagerLogReader, which is a disk operation. Currently, this happens in the same thread as the Persister which means that other share coordinator requests remain queued on a potentially slow disk read operation, again choking the persister.
Fix: Allocate separate executor to the record readers. 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)