You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "apoorvmittal10 (via GitHub)" <gi...@apache.org> on 2024/09/04 20:36:09 UTC
[PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
apoorvmittal10 opened a new pull request, #17097:
URL: https://github.com/apache/kafka/pull/17097
The PR introduces states for share partition which are used to make share partition initilzation async.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17097:
URL: https://github.com/apache/kafka/pull/17097#discussion_r1746387937
##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -626,6 +613,85 @@ void maybeProcessFetchQueue() {
}
}
+ private SharePartition fetchSharePartition(SharePartitionKey sharePartitionKey) {
+ return partitionCacheMap.computeIfAbsent(sharePartitionKey,
+ k -> {
+ long start = time.hiResClockMs();
+ SharePartition partition = new SharePartition(
+ sharePartitionKey.groupId,
+ sharePartitionKey.topicIdPartition,
+ maxInFlightMessages,
+ maxDeliveryCount,
+ recordLockDurationMs,
+ timer,
+ time,
+ persister
+ );
+ this.shareGroupMetrics.partitionLoadTime(start);
+ return partition;
+ });
+ }
+
+ private void maybeCompleteInitializationWithException(
+ SharePartitionKey sharePartitionKey,
+ CompletableFuture<Map<TopicIdPartition, PartitionData>> future,
+ Throwable throwable) {
+ if (throwable instanceof LeaderNotAvailableException) {
+ log.debug("The share partition with key {} is not initialized yet", sharePartitionKey);
+ // Do not process the fetch request for this partition as the leader is not initialized yet.
+ // The fetch request will be retried in the next poll.
+ // TODO: Add the request to delayed fetch purgatory.
+ return;
+ }
+
+ if (throwable instanceof NotLeaderOrFollowerException || throwable instanceof FencedStateEpochException) {
+ log.info("The share partition with key {} is fenced with message: {}", sharePartitionKey, throwable.getMessage());
Review Comment:
nit: I'd remove the "with message". Still makes sense that you're adding the exception message after the colon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17097:
URL: https://github.com/apache/kafka/pull/17097#discussion_r1749872314
##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -626,6 +613,85 @@ void maybeProcessFetchQueue() {
}
}
+ private SharePartition fetchSharePartition(SharePartitionKey sharePartitionKey) {
+ return partitionCacheMap.computeIfAbsent(sharePartitionKey,
+ k -> {
+ long start = time.hiResClockMs();
+ SharePartition partition = new SharePartition(
+ sharePartitionKey.groupId,
+ sharePartitionKey.topicIdPartition,
+ maxInFlightMessages,
+ maxDeliveryCount,
+ recordLockDurationMs,
+ timer,
+ time,
+ persister
+ );
+ this.shareGroupMetrics.partitionLoadTime(start);
+ return partition;
+ });
+ }
+
+ private void maybeCompleteInitializationWithException(
+ SharePartitionKey sharePartitionKey,
+ CompletableFuture<Map<TopicIdPartition, PartitionData>> future,
+ Throwable throwable) {
+ if (throwable instanceof LeaderNotAvailableException) {
+ log.debug("The share partition with key {} is not initialized yet", sharePartitionKey);
+ // Do not process the fetch request for this partition as the leader is not initialized yet.
+ // The fetch request will be retried in the next poll.
+ // TODO: Add the request to delayed fetch purgatory.
+ return;
+ }
+
+ if (throwable instanceof NotLeaderOrFollowerException || throwable instanceof FencedStateEpochException) {
+ log.info("The share partition with key {} is fenced with message: {}", sharePartitionKey, throwable.getMessage());
Review Comment:
Thanks, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
Posted by "omkreddy (via GitHub)" <gi...@apache.org>.
omkreddy merged PR #17097:
URL: https://github.com/apache/kafka/pull/17097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on PR #17097:
URL: https://github.com/apache/kafka/pull/17097#issuecomment-2337564195
@AndrewJSchofield @omkreddy Build passed with unrelated test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-17482: Make share partition initialization async (KIP-932) [kafka]
Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on PR #17097:
URL: https://github.com/apache/kafka/pull/17097#issuecomment-2331108556
@AndrewJSchofield @adixitconfluent @omkreddy Build passed with unrelated tests failure. Can you please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org