[jira] [Created] (KAFKA-14663) High throughput topics can starve low-throughput MM2 offset syncs

["Greg Harris (Jira)" <ji...@apache.org>]

Greg Harris created KAFKA-14663:
-----------------------------------

             Summary: High throughput topics can starve low-throughput MM2 offset syncs
                 Key: KAFKA-14663
                 URL: https://issues.apache.org/jira/browse/KAFKA-14663
             Project: Kafka
          Issue Type: Bug
          Components: mirrormaker
    Affects Versions: 3.3.0, 3.0.0, 3.1.0, 3.4.0, 3.5.0
            Reporter: Greg Harris
            Assignee: Greg Harris


In MM2, a semaphore is used to throttle the number of offset syncs written to the offset-syncs topic. If too many offset writes are requested (for example, from high-throughput topics) then some are silently dropped and never retried. This is acceptable for a single topic-partition, where a later record may re-trigger the offset-sync and write the sync successfully.

However, if there is a large variance between throughput in the topics emitted by an MM2 instance, it is possible for high-throughput topics to trigger many offset syncs, and cause the offset-syncs for a co-located low-throughput topic to be unfairly dropped.

This can cause the downstream offsets for the starved topic to lag behind significantly, or be prevented completely.

Instead, we should have some sort of fairness mechanism where low-thoughput topics are given similar priority to high-throughput topics in producing offset syncs, and cause excess sync messages from high-throughput topics to be dropped instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)