You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "David Smiley (Jira)" <ji...@apache.org> on 2026/03/21 03:07:00 UTC

[jira] [Reopened] (SOLR-18077) CrossDC Consumer - out-of-order Kafka partition processing

     [ https://issues.apache.org/jira/browse/SOLR-18077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Smiley reopened SOLR-18077:
---------------------------------

> CrossDC Consumer - out-of-order Kafka partition processing
> ----------------------------------------------------------
>
>                 Key: SOLR-18077
>                 URL: https://issues.apache.org/jira/browse/SOLR-18077
>             Project: Solr
>          Issue Type: Bug
>          Components: module - crossDC
>    Affects Versions: 9.10
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: main(11.0), 10.1
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> When mirrored requests are submitted to Kafka in {{KafkaMirroringSink}} the default partitioner is used (\{{BuiltInPartitioner}}), which is submits messages to partitions in batches, switching between partitions in a round-robin fashion.
> The same partitioner will be used (see below) by the MirrorMaker when adding messages to the target Kafka topic. Then {{KafkaCrossDcConsumer.pollAndProcessRequests()}} method retrieves new records - BUT then it iterates over partitions in a basically random order because {{ConsumerRecords.partitions}} is a HashMap.
> This means that the batches of messages retrieved from multiple partitions are no longer necessarily in the same order as they were submitted. If requests in these batches from multiple partitions refer to the same collection then they may be applied out of order, leading to data divergence.
> One possible solution is to explicitly use a different partitioning scheme when submitting messages from {{KafkaMirroringSink}} . This happens automatically when {{ProducerRecord}} key is explicitly set, and we can use the {{collection}} name as the key - this way all requests for the same collection will end up in the same partition, thus preserving the ordering.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org