[jira] [Created] (SOLR-18408) CrossDC Consumer: incorrect nextOffset commits

["Andrzej Bialecki (Jira)" <ji...@apache.org>]

Andrzej Bialecki created SOLR-18408:
---------------------------------------

             Summary: CrossDC Consumer: incorrect nextOffset commits
                 Key: SOLR-18408
                 URL: https://issues.apache.org/jira/browse/SOLR-18408
             Project: Solr
          Issue Type: Bug
          Components: module - crossDC
    Affects Versions: 9.10.1, 10.0
            Reporter: Andrzej Bialecki
            Assignee: Andrzej Bialecki


{{PartitionManager.getOffsetForPartition}} returns {{partitionRecords.get(partitionRecords.size() - 1).offset() + 1}}
- the last offset of the _entire_ poll batch for that partition. Every {{WorkUnit}} created while
iterating that batch therefore receives the same {{{}nextOffset{}}}. The first WorkUnit to complete commits an offset past all the later WorkUnits that are still in flight.

Example failure scenario: a poll returns records at offsets 100–199 for one partition and the collapse logic splits them into three WorkUnits (e.g. because some records contained only additions and some others contained also deletes, which enforces a buffer flush + new WorkUnit). All three units now carry {{{}nextOffset = 200{}}}. The first completes and
commits 200. The consumer crashes while the other two are still executing. On restart the consumer resumes at 200 and the records in the two unfinished WorkUnits are never replayed. Data is lost.

Additionally, {{PartitionManager.checkForOffsetUpdates}} synchronizes on the method argument {{TopicPartition}} which is equal-but-distinct object returned by Kafka from different threads, so the synchronization doesn't work.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org