You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ShivsundarR (via GitHub)" <gi...@apache.org> on 2025/10/22 17:26:17 UTC

[PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

ShivsundarR opened a new pull request, #20752:
URL: https://github.com/apache/kafka/pull/20752

   *What*
   https://issues.apache.org/jira/browse/KAFKA-19789
   
   - There were some scenarios where `ShareFetchResponse` contained duplicate acquired records, this was a broker side bug.
   - Although ideally this should not happen, the client was not expecting this case and acknowledged with `GAP` type for any duplicate occurrence.
   - This case should be logged as an error in the client, and we must not acknowledge the duplicate offsets as the broker is already in a bad state.
   - PR adds an error log for this case and a unit test for the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield merged PR #20752:
URL: https://github.com/apache/kafka/pull/20752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2453054436


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -100,9 +100,16 @@ public class ShareCompletedFetch {
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
         List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Set to find duplicates in case of overlapping acquired records
+        Set<Long> offsets = new HashSet<>();
         partitionAcquiredRecords.forEach(acquiredRecords -> {
             for (long offset = acquiredRecords.firstOffset(); offset <= acquiredRecords.lastOffset(); offset++) {
-                acquiredRecordList.add(new OffsetAndDeliveryCount(offset, acquiredRecords.deliveryCount()));
+                if (!offsets.add(offset)) {
+                    log.error("Duplicate acquired record offset {} found in share fetch response for partition {}. " +
+                            "This indicates a broker processing issue.", offset, partition.topicPartition());

Review Comment:
   Just curious, are there any known issues that lead to duplicate offsets?



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetchTest.java:
##########
@@ -356,6 +357,63 @@ record = records.get(1);
         assertEquals(0, records.size());
     }
 
+    @Test
+    public void testOverlappingAcquiredRecordsLogsErrorAndRetainsFirstOccurrence() {
+        int startingOffset = 0;
+        int numRecords = 20;        // Records for 0-19
+
+        // Create overlapping acquired records: [0-9] and [5-14]
+        // Offsets 5-9 will be duplicates
+        List<ShareFetchResponseData.AcquiredRecords> acquiredRecords = new ArrayList<>();
+        acquiredRecords.add(new ShareFetchResponseData.AcquiredRecords()
+                .setFirstOffset(0L)
+                .setLastOffset(9L)
+                .setDeliveryCount((short) 1));
+        acquiredRecords.add(new ShareFetchResponseData.AcquiredRecords()
+                .setFirstOffset(5L)
+                .setLastOffset(14L)
+                .setDeliveryCount((short) 2));
+
+        ShareFetchResponseData.PartitionData partitionData = new ShareFetchResponseData.PartitionData()
+                .setRecords(newRecords(startingOffset, numRecords))
+                .setAcquiredRecords(acquiredRecords);
+
+        ShareCompletedFetch completedFetch = newShareCompletedFetch(partitionData);
+
+        Deserializers<String, String> deserializers = newStringDeserializers();
+
+        // Fetch records and verify that only 15 unique records are returned (0-14)
+        ShareInFlightBatch<String, String> batch = completedFetch.fetchRecords(deserializers, 20, true);
+        List<ConsumerRecord<String, String>> records = batch.getInFlightRecords();
+        
+        // Should get 15 unique records: 0-9 from first range (with deliveryCount=1)
+        // and 10-14 from second range (with deliveryCount=2)
+        assertEquals(15, records.size());
+        
+        // Verify first occurrence (offset 5 should have deliveryCount=1 from first range)
+        ConsumerRecord<String, String> record5 = records.stream()
+                .filter(r -> r.offset() == 5L)
+                .findFirst()
+                .orElse(null);
+        assertNotNull(record5);
+        assertEquals(Optional.of((short) 1), record5.deliveryCount());
+        
+        // Verify offset 10 has deliveryCount=2 from second range
+        ConsumerRecord<String, String> record10 = records.stream()
+                .filter(r -> r.offset() == 10L)
+                .findFirst()
+                .orElse(null);
+        assertNotNull(record10);
+        assertEquals(Optional.of((short) 2), record10.deliveryCount());
+        
+        // Verify all offsets are unique
+        Set<Long> offsetSet = new HashSet<>();
+        for (ConsumerRecord<String, String> record : records) {

Review Comment:
   I'm not sure if this covers the new behavior, since `inFlightRecords` already handles offset deduplication.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2459621417


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -99,10 +99,21 @@ public class ShareCompletedFetch {
     }
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
-        List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Setting the size of the array to the size of the first batch of acquired records. In case there is only 1 batch acquired, resizing would not happen.
+        int initialListSize = !partitionAcquiredRecords.isEmpty() ? (int) (partitionAcquiredRecords.get(0).lastOffset() -

Review Comment:
   In the case where the `partitionAcquiredRecords` is empty, we can just make an empty list and return directly. We don't need to make the `HashSet` only to discard it unused because the loop will not have any iterations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2454582774


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -100,9 +100,16 @@ public class ShareCompletedFetch {
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
         List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Set to find duplicates in case of overlapping acquired records
+        Set<Long> offsets = new HashSet<>();

Review Comment:
   I wonder if you could change the `partitionAcquiredRecords` into a `LinkedHashMap` or similar to combine the duplicate checking with the ordered iteration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2454044102


##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetchTest.java:
##########
@@ -356,6 +357,63 @@ record = records.get(1);
         assertEquals(0, records.size());
     }
 
+    @Test
+    public void testOverlappingAcquiredRecordsLogsErrorAndRetainsFirstOccurrence() {
+        int startingOffset = 0;
+        int numRecords = 20;        // Records for 0-19
+
+        // Create overlapping acquired records: [0-9] and [5-14]
+        // Offsets 5-9 will be duplicates
+        List<ShareFetchResponseData.AcquiredRecords> acquiredRecords = new ArrayList<>();
+        acquiredRecords.add(new ShareFetchResponseData.AcquiredRecords()
+                .setFirstOffset(0L)
+                .setLastOffset(9L)
+                .setDeliveryCount((short) 1));
+        acquiredRecords.add(new ShareFetchResponseData.AcquiredRecords()
+                .setFirstOffset(5L)
+                .setLastOffset(14L)
+                .setDeliveryCount((short) 2));
+
+        ShareFetchResponseData.PartitionData partitionData = new ShareFetchResponseData.PartitionData()
+                .setRecords(newRecords(startingOffset, numRecords))
+                .setAcquiredRecords(acquiredRecords);
+
+        ShareCompletedFetch completedFetch = newShareCompletedFetch(partitionData);
+
+        Deserializers<String, String> deserializers = newStringDeserializers();
+
+        // Fetch records and verify that only 15 unique records are returned (0-14)
+        ShareInFlightBatch<String, String> batch = completedFetch.fetchRecords(deserializers, 20, true);
+        List<ConsumerRecord<String, String>> records = batch.getInFlightRecords();
+        
+        // Should get 15 unique records: 0-9 from first range (with deliveryCount=1)
+        // and 10-14 from second range (with deliveryCount=2)
+        assertEquals(15, records.size());
+        
+        // Verify first occurrence (offset 5 should have deliveryCount=1 from first range)
+        ConsumerRecord<String, String> record5 = records.stream()
+                .filter(r -> r.offset() == 5L)
+                .findFirst()
+                .orElse(null);
+        assertNotNull(record5);
+        assertEquals(Optional.of((short) 1), record5.deliveryCount());
+        
+        // Verify offset 10 has deliveryCount=2 from second range
+        ConsumerRecord<String, String> record10 = records.stream()
+                .filter(r -> r.offset() == 10L)
+                .findFirst()
+                .orElse(null);
+        assertNotNull(record10);
+        assertEquals(Optional.of((short) 2), record10.deliveryCount());
+        
+        // Verify all offsets are unique
+        Set<Long> offsetSet = new HashSet<>();
+        for (ConsumerRecord<String, String> record : records) {

Review Comment:
   Yes the logic around `inFlightRecords` ensures we do not send duplicate offsets to the application side, but the client does respond with a GAP acknowledgement to the broker for any duplicate offset.
   https://github.com/apache/kafka/blob/2c444485123a5790170856284f613384b4b033c2/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java#L219
   
   Without deduplication, when the offset is encountered second time,`lastRecord.offset > nextAcquired.offset`, (as `nextAcquired` will be an older offset) will be true, so the client acknowledges these offsets as GAPs which is kind of hiding the main issue.
   As the broker is already in a bad state(duplication should never happen), we thought of logging an error and ignoring any duplicates on the client.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2459012672


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -99,10 +99,17 @@ public class ShareCompletedFetch {
     }
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
-        List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        List<OffsetAndDeliveryCount> acquiredRecordList = new ArrayList<>();

Review Comment:
   That makes sense, if most of the times the response is gonna contain only 1 batch, we can avoid resizing. I have made the change. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2459723779


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -99,10 +99,21 @@ public class ShareCompletedFetch {
     }
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
-        List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Setting the size of the array to the size of the first batch of acquired records. In case there is only 1 batch acquired, resizing would not happen.
+        int initialListSize = !partitionAcquiredRecords.isEmpty() ? (int) (partitionAcquiredRecords.get(0).lastOffset() -

Review Comment:
   Yes makes sense, I have updated the code now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2454542490


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -100,9 +100,16 @@ public class ShareCompletedFetch {
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
         List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Set to find duplicates in case of overlapping acquired records
+        Set<Long> offsets = new HashSet<>();
         partitionAcquiredRecords.forEach(acquiredRecords -> {
             for (long offset = acquiredRecords.firstOffset(); offset <= acquiredRecords.lastOffset(); offset++) {
-                acquiredRecordList.add(new OffsetAndDeliveryCount(offset, acquiredRecords.deliveryCount()));
+                if (!offsets.add(offset)) {
+                    log.error("Duplicate acquired record offset {} found in share fetch response for partition {}. " +
+                            "This indicates a broker processing issue.", offset, partition.topicPartition());

Review Comment:
   There are no current known issues, but there was previously an issue in the broker and adding logging would have made it quicker to get to the bottom of it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2454861739


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -100,9 +100,16 @@ public class ShareCompletedFetch {
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
         List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Set to find duplicates in case of overlapping acquired records
+        Set<Long> offsets = new HashSet<>();

Review Comment:
   I had a look into making the `acquiredRecordsList(LinkedList<OffsetAndDeliveryCount>)` into a `LinkedHashMap` This change would actually have a bit of a code change around `listIterator`, we might have to use `map.entrySet().iterator()` for rewinding to the start of the list.
   And as we are doing sequential operations and not key based, probably better to keep it as a list?
   I have changed it to `ArrayList` instead of a `LinkedList` though as it would give better iteration performance for build once and iterate use cases.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2454044648


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -100,9 +100,16 @@ public class ShareCompletedFetch {
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
         List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        // Set to find duplicates in case of overlapping acquired records
+        Set<Long> offsets = new HashSet<>();
         partitionAcquiredRecords.forEach(acquiredRecords -> {
             for (long offset = acquiredRecords.firstOffset(); offset <= acquiredRecords.lastOffset(); offset++) {
-                acquiredRecordList.add(new OffsetAndDeliveryCount(offset, acquiredRecords.deliveryCount()));
+                if (!offsets.add(offset)) {
+                    log.error("Duplicate acquired record offset {} found in share fetch response for partition {}. " +
+                            "This indicates a broker processing issue.", offset, partition.topicPartition());

Review Comment:
   Yes, there was a broker side issue when `SharePartition` was at capacity - https://issues.apache.org/jira/browse/KAFKA-19808. Due to this, we were getting duplicate offsets (with different delivery counts) in the `ShareFetchResponse`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-19789: Log an error when we get duplicate acquired offsets in ShareFetchResponse. [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #20752:
URL: https://github.com/apache/kafka/pull/20752#discussion_r2455965933


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareCompletedFetch.java:
##########
@@ -99,10 +99,17 @@ public class ShareCompletedFetch {
     }
 
     private List<OffsetAndDeliveryCount> buildAcquiredRecordList(List<ShareFetchResponseData.AcquiredRecords> partitionAcquiredRecords) {
-        List<OffsetAndDeliveryCount> acquiredRecordList = new LinkedList<>();
+        List<OffsetAndDeliveryCount> acquiredRecordList = new ArrayList<>();

Review Comment:
   By default, a new list will have space for 10 elements. Resizing is expensive. Maybe one optimisation would be to see how many offsets are in the first element in the `partitionAcquiredRecords`, and using that number as the initial size of the list. In the case of only one batch of offsets, the list will be the correct size alrready. wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org