[jira] [Resolved] (KAFKA-14663) High throughput topics can starve low-throughput MM2 offset syncs

["Chris Egerton (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton resolved KAFKA-14663.
-----------------------------------
    Resolution: Duplicate

> High throughput topics can starve low-throughput MM2 offset syncs
> -----------------------------------------------------------------
>
>                 Key: KAFKA-14663
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14663
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 3.1.0, 3.0.0, 3.3.0, 3.4.0, 3.5.0
>            Reporter: Greg Harris
>            Assignee: Greg Harris
>            Priority: Major
>
> In MM2, a semaphore is used to throttle the number of offset syncs written to the offset-syncs topic. If too many offset writes are requested (for example, from high-throughput topics) then some are silently dropped and never retried. This is acceptable for a single topic-partition, where a later record may re-trigger the offset-sync and write the sync successfully.
> However, if there is a large variance between throughput in the topics emitted by an MM2 instance, it is possible for high-throughput topics to trigger many offset syncs, and cause the offset-syncs for a co-located low-throughput topic to be unfairly dropped.
> This can cause the downstream offsets for the starved topic to lag behind significantly, or be prevented completely.
> Instead, we should have some sort of fairness mechanism where low-thoughput topics are given similar priority to high-throughput topics in producing offset syncs, and cause excess sync messages from high-throughput topics to be dropped instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)