You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "apoorvmittal10 (via GitHub)" <gi...@apache.org> on 2025/12/02 14:57:28 UTC
[PR] KAFKA-19954: Fixing overlapping batches post re-initalization of Share Partition [kafka]
apoorvmittal10 opened a new pull request, #21051:
URL: https://github.com/apache/kafka/pull/21051
The PR fixes the issue when SharePartition is re-initalized and starts fetch prior to start offset. In edge cases there occurs a scenario when Acquire method creates overlapping batches in cache. The issue is more evident in `record_limit` mode.
Consider a scenario where log has 0-99 offsets as batch and post re-initialization of share partition the start offset becomes 5, first 5 records already acknowledged in previous share-partition instance. Prior to the fix:
- The request arrives for fetching next 5 offsets hence the cache will hold 5-99, with 5-9 offsets as acquired.
- Client acknowledes 5-9 offset, start offset moves to 10.
- Client re-fetches from offset 10 and gets same 0-99 batch.
- Acquire readjusts the base offset to 10 as start offset has moved.
- There won't be any overlapping batch in cache for 10-99 as the key in cache is 5.
- Hence, 0-99 cache entry will be created
Post Fix:
- There will be an overlapping entry in the cache as start considering the minimum of baseOffset and batch's firstOffset.
Also fixed other scenarios where due to startOffset move and no cache overlap the offsets could be acquired prior to startOffset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19954: Fixing overlapping batches post re-initalization of Share Partition [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 merged PR #21051:
URL: https://github.com/apache/kafka/pull/21051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19954: Fixing overlapping batches post re-initalization of Share Partition [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #21051:
URL: https://github.com/apache/kafka/pull/21051#discussion_r2582339273
##########
core/src/test/java/kafka/server/share/SharePartitionTest.java:
##########
@@ -10635,6 +10738,113 @@ public void testAcquireMultipleRecordsWithOverlapAndNewBatchInRecordLimitMode()
assertEquals(5, sharePartitionMetrics.inFlightBatchMessageCount().max());
}
+ /**
+ * Test validates the scenario where the partition has been re-initialized with a moved start offset.
+ * In this case, the acquire logic should respect the new start offset and not allow acquiring
+ * records post the new start offset. For the test, simulate a scenario where the log batch is 0-99
Review Comment:
Done, my bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19954: Fixing overlapping batches post re-initalization of Share Partition [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #21051:
URL: https://github.com/apache/kafka/pull/21051#discussion_r2582065371
##########
core/src/test/java/kafka/server/share/SharePartitionTest.java:
##########
@@ -10635,6 +10738,113 @@ public void testAcquireMultipleRecordsWithOverlapAndNewBatchInRecordLimitMode()
assertEquals(5, sharePartitionMetrics.inFlightBatchMessageCount().max());
}
+ /**
+ * Test validates the scenario where the partition has been re-initialized with a moved start offset.
+ * In this case, the acquire logic should respect the new start offset and not allow acquiring
+ * records post the new start offset. For the test, simulate a scenario where the log batch is 0-99
Review Comment:
nit: I think "records before the new start offset" is what it means. The fix makes sure the earliest acquisition offset is never before the start offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org