You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "chirag-wadhwa5 (via GitHub)" <gi...@apache.org> on 2025/01/23 12:21:36 UTC
[PR] KAFKA-18494-2: Bugfix in sharePartition. Added gapsAfterInitialization that store the first offsets of all gaps during share partition initialization [kafka]
chirag-wadhwa5 opened a new pull request, #18677:
URL: https://github.com/apache/kafka/pull/18677
Bug -> there are times during share partition initialization, where the persister read state response contains batches, eg -
startOffset = 10, stateBatches = {{15, 20}, {30, 40}}
These gaps could natural gaps, but could also be batches which were fetched before but were not acknowledged. In that case, these records must be made available again for fetch.
This PR introduces another class variable, `gapsAfterInitialization`, which stores the first offset of all these gaps in the read state response. The `nextFetchOffset `method then consider these gaps as well while finding the next offset to fetch from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18494-2: Bugfix in sharePartition. Added gapsAfterInitialization that store the first offsets of all gaps during share partition initialization [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 closed pull request #18677: KAFKA-18494-2: Bugfix in sharePartition. Added gapsAfterInitialization that store the first offsets of all gaps during share partition initialization
URL: https://github.com/apache/kafka/pull/18677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18494-2: Bugfix in sharePartition. Added gapsAfterInitialization that store the first offsets of all gaps during share partition initialization [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on PR #18677:
URL: https://github.com/apache/kafka/pull/18677#issuecomment-2616116307
I have closed this PR as we have a new Pr for this bug now: https://github.com/apache/kafka/pull/18696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-18494-2: Bugfix in sharePartition. Added gapsAfterInitialization that store the first offsets of all gaps during share partition initialization [kafka]
Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #18677:
URL: https://github.com/apache/kafka/pull/18677#discussion_r1926963735
##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -287,6 +288,12 @@ public static RecordState forId(byte id) {
*/
private long endOffset;
+ /**
+ * This maintains the set of gaps in the cachedState after initialization. Some of these gaps might correspond to batches
+ * which were fetched but not acknowledged yet. The gaps are used in determining the next fetch offset.
+ */
+ private final TreeSet<Long> gapsAfterInitialization;
Review Comment:
Is TreeSet necessary (`lg(n)` time data structure).
You are assured that the readState reponse will always be sorted on the intervals - the order is
firstOffset, lastOffset, deliveryCount, deliveryState
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org