You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ShivsundarR (via GitHub)" <gi...@apache.org> on 2024/07/30 07:58:46 UTC

[PR] KAFKA-17217: Batched acknowledge requests per node [kafka]

ShivsundarR opened a new pull request, #16727:
URL: https://github.com/apache/kafka/pull/16727

   In ShareConsumeRequestManager, currently every time we perform a `commitSync`/`commitAsync`/`acknowledgeOnClose`  we create one AcknowledgeRequestState for each call. But this can be optimised further as we can batch up the acknowledgements to be sent to the same node. 
   We can optimise the number of RPC calls by batching the acknowledgements before the next poll is invoked per node. 
   This will ensure that between 2 polls, the acknowledgements are accumulated in one request per node and then sent during poll, resulting in lesser RPC calls.
   We are storing a pair of acknowledge request states for every node, the first value denotes the requestState for commitAsync() and the second value denotes the requestState for commitSync() and acknowledgeOnClose().


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1705254620


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -520,31 +588,38 @@ private void handleShareAcknowledgeSuccess(Node fetchTarget,
                 acknowledgeRequestState.onFailedAttempt(currentTimeMs);
                 if (response.error().exception() instanceof RetriableException && !closing) {
                     // We retry the request until the timer expires, unless we are closing.
-                } else {
-                    requestData.topics().forEach(topic -> topic.partitions().forEach(partition -> {
-                        TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                                partition.partitionIndex(),
-                                metadata.topicNames().get(topic.topicId()));
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                        acknowledgeRequestState.handleAcknowledgeErrorCode(tip, response.error());
-                    }));
-
-                    acknowledgeRequestState.processingComplete();
+                    acknowledgeRequestState.retryRequest();

Review Comment:
   Yes, thanks had missed that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698096040


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -50,20 +50,21 @@
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
-import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
-import java.util.Queue;
 import java.util.Set;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.BiConsumer;
 import java.util.stream.Collectors;
 
 /**
- * {@code ShareConsumeRequestManager} is responsible for generating {@link ShareFetchRequest} and
- * {@link ShareAcknowledgeRequest} to fetch and acknowledge records being delivered for a consumer
+ * {@code ShareConsumeRequestManager} is responsible for generating

Review Comment:
   nit: Comment formatting messed up without changing the works at all.



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManagerTest.java:
##########
@@ -1022,9 +1194,12 @@ private List<UnsentRequest> removeUnsentRequestByNode(Node node) {
 
         @Override
         protected void checkDisconnects(final long currentTimeMs) {
-            // any disconnects affecting requests that have already been transmitted will be handled
-            // by NetworkClient, so we just need to check whether connections for any of the unsent
-            // requests have been disconnected; if they have, then we complete the corresponding future
+            // any disconnects affecting requests that have already been transmitted will be

Review Comment:
   formatting



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());
+            }
+        }
+        if (!acknowledgeRequestStates.isEmpty())
+            areAnyAcksLeft = true;
+        return areAnyAcksLeft;
+    }
+
+    private boolean isRequestStateEmpty(AcknowledgeRequestState acknowledgeRequestState) {
+        return acknowledgeRequestState == null
+                || !acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                || !acknowledgeRequestState.incompleteAcknowledgements.isEmpty()
+                || !acknowledgeRequestState.inFlightAcknowledgements.isEmpty();
+    }
+
     /**
      * Enqueue an AcknowledgeRequestState to be picked up on the next poll
      *
      * @param acknowledgementsMap The acknowledgements to commit
-     * @param deadlineMs          Time until which the request will be retried if it fails with
+     * @param deadlineMs          Time until which the request will be retried if it

Review Comment:
   Formatting



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -449,7 +531,8 @@ private void handleShareFetchSuccess(Node fetchTarget,
                     }
                     acks.setAcknowledgeErrorCode(Errors.forCode(partitionData.acknowledgeErrorCode()));
                     Map<TopicIdPartition, Acknowledgements> acksMap = Collections.singletonMap(tip, acks);
-                    ShareAcknowledgementCommitCallbackEvent event = new ShareAcknowledgementCommitCallbackEvent(acksMap);
+                    ShareAcknowledgementCommitCallbackEvent event = new ShareAcknowledgementCommitCallbackEvent(

Review Comment:
   If you want to make the line shorter, it's more readable to break it before "new".



##########
clients/src/test/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManagerTest.java:
##########
@@ -1034,7 +1209,8 @@ protected void checkDisconnects(final long currentTimeMs) {
                         FutureCompletionHandler handler = unsentRequest.handler();
                         AuthenticationException authenticationException = client.authenticationException(node);
                         long startMs = unsentRequest.timer().currentTimeMs() - unsentRequest.timer().elapsedMs();
-                        handler.onComplete(new ClientResponse(makeHeader(unsentRequest.requestBuilder().latestAllowedVersion()),
+                        handler.onComplete(new ClientResponse(
+                                makeHeader(unsentRequest.requestBuilder().latestAllowedVersion()),

Review Comment:
   nit: This would be neater with the `new ClientResponse(` on the new line in its entirety.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -207,46 +212,32 @@ public void fetch(Map<TopicIdPartition, Acknowledgements> acknowledgementsMap) {
     }
 
     /**
-     * Process acknowledgeRequestStates and prepares a list of acknowledgements to be sent in the poll().
+     * Process acknowledgeRequestStates and prepares a list of acknowledgements to
+     * be sent in the poll().
      *
      * @param currentTimeMs the current time in ms.
      *
      * @return the PollResult containing zero or more acknowledgements.
      */
     private PollResult processAcknowledgements(long currentTimeMs) {
         List<UnsentRequest> unsentRequests = new ArrayList<>();
-        Iterator<AcknowledgeRequestState> iterator = acknowledgeRequestStates.iterator();
-        while (iterator.hasNext()) {
-            AcknowledgeRequestState acknowledgeRequestState = iterator.next();
-            if (acknowledgeRequestState.isProcessed()) {
-                iterator.remove();
-            } else if (!acknowledgeRequestState.maybeExpire()) {
-                if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
-                    log.trace("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
-                } else {
-                    if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
-                        acknowledgeRequestState.onSendAttempt(currentTimeMs);
-                        UnsentRequest request = acknowledgeRequestState.buildRequest(currentTimeMs);
-                        if (request != null) {
-                            unsentRequests.add(request);
-                        }
-                    }
-                }
-            } else {
-                // Fill in TimeoutException
-                for (TopicIdPartition tip : acknowledgeRequestState.acknowledgementsMap.keySet()) {
-                    metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                    acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.REQUEST_TIMED_OUT);
-                }
-                iterator.remove();
-            }
+        for (Pair<AcknowledgeRequestState> requestStates : acknowledgeRequestStates.values()) {
+            isAsyncDone = false;
+            // For commitAsync
+            maybeBuildRequest(requestStates.getFirst(), currentTimeMs, true).ifPresent(unsentRequests::add);
+
+            // Check to ensure we start processing commitSync/close only if there are no
+            // commitAsync requests left to process.
+            if (isAsyncDone)
+                maybeBuildRequest(requestStates.getSecond(), currentTimeMs, false).ifPresent(unsentRequests::add);
         }
 
         PollResult pollResult = null;
         if (!unsentRequests.isEmpty()) {
             pollResult = new PollResult(unsentRequests);
-        } else if (!acknowledgeRequestStates.isEmpty()) {
-            // Return empty result until all the acknowledgement request states are processed
+        } else if (areAnyAcknowledgementsLeft()) {
+            // Return empty result until all the acknowledgement request states are

Review Comment:
   formatting



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -50,20 +50,21 @@
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashMap;
-import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
-import java.util.Queue;
 import java.util.Set;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.BiConsumer;
 import java.util.stream.Collectors;
 
 /**
- * {@code ShareConsumeRequestManager} is responsible for generating {@link ShareFetchRequest} and
- * {@link ShareAcknowledgeRequest} to fetch and acknowledge records being delivered for a consumer
+ * {@code ShareConsumeRequestManager} is responsible for generating

Review Comment:
   nit: Comment formatting messed up without changing the words at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709359480


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -518,33 +612,48 @@ private void handleShareAcknowledgeSuccess(Node fetchTarget,
 
             if (!handler.handleResponse(response, requestVersion)) {
                 acknowledgeRequestState.onFailedAttempt(currentTimeMs);
-                if (response.error().exception() instanceof RetriableException && !closing) {
+                if (response.error().exception() instanceof RetriableException && !acknowledgeRequestState.onClose) {
                     // We retry the request until the timer expires, unless we are closing.
+                    acknowledgeRequestState.retryRequest();
                 } else {
-                    requestData.topics().forEach(topic -> topic.partitions().forEach(partition -> {
-                        TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                                partition.partitionIndex(),
-                                metadata.topicNames().get(topic.topicId()));
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                    response.data().responses().forEach(shareAcknowledgeTopicResponse -> shareAcknowledgeTopicResponse.partitions().forEach(partitionData -> {
+                        TopicIdPartition tip = new TopicIdPartition(shareAcknowledgeTopicResponse.topicId(),
+                                partitionData.partitionIndex(),
+                                metadata.topicNames().get(shareAcknowledgeTopicResponse.topicId()));
+
                         acknowledgeRequestState.handleAcknowledgeErrorCode(tip, response.error());
+                        metricsManager.recordLatency(resp.requestLatencyMs());
                     }));
-
-                    acknowledgeRequestState.processingComplete();
                 }
             } else {
-                response.data().responses().forEach(topic -> topic.partitions().forEach(partition -> {
-                    TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                            partition.partitionIndex(),
-                            metadata.topicNames().get(topic.topicId()));
-                    if (partition.errorCode() != Errors.NONE.code()) {
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                AtomicBoolean shouldRetry = new AtomicBoolean(false);

Review Comment:
   I had it made it `AtomicBoolean` as we are trying to modify it inside a lambda expression. It did not allow `boolean`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709303701


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -749,58 +869,104 @@ UnsentRequest buildRequest(long currentTimeMs) {
 
             BiConsumer<ClientResponse, Throwable> responseHandler = (clientResponse, error) -> {
                 if (error != null) {
-                    onFailedAttempt(currentTimeMs);
                     errorHandler.handle(nodeToSend, requestBuilder.data(), this, error, currentTimeMs);
                     processingComplete();
                 } else {
                     successHandler.handle(nodeToSend, requestBuilder.data(), this, clientResponse, currentTimeMs);
+                    if (onClose && !closeFuture.isDone()) {
+                        closeFuture.complete(null);
+                    }
                 }
             };
 
             if (requestBuilder == null) {
-                log.trace("Building ShareAcknowledge request to send to node {} failed", nodeToSend.id());
                 handleSessionErrorCode(Errors.SHARE_SESSION_NOT_FOUND);
                 return null;
             } else {
-                log.trace("Building ShareAcknowledge request to send to node {}", nodeToSend.id());
+                inFlightAcknowledgements.putAll(finalAcknowledgementsToSend);
+                if (incompleteAcknowledgements.isEmpty()) {
+                    acknowledgementsToSend.clear();
+                } else {
+                    incompleteAcknowledgements.clear();

Review Comment:
   We do move the acknowledgements to the inFlightAcknowledgements right, so we clear the map that we moved it from.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709303701


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -749,58 +869,104 @@ UnsentRequest buildRequest(long currentTimeMs) {
 
             BiConsumer<ClientResponse, Throwable> responseHandler = (clientResponse, error) -> {
                 if (error != null) {
-                    onFailedAttempt(currentTimeMs);
                     errorHandler.handle(nodeToSend, requestBuilder.data(), this, error, currentTimeMs);
                     processingComplete();
                 } else {
                     successHandler.handle(nodeToSend, requestBuilder.data(), this, clientResponse, currentTimeMs);
+                    if (onClose && !closeFuture.isDone()) {
+                        closeFuture.complete(null);
+                    }
                 }
             };
 
             if (requestBuilder == null) {
-                log.trace("Building ShareAcknowledge request to send to node {} failed", nodeToSend.id());
                 handleSessionErrorCode(Errors.SHARE_SESSION_NOT_FOUND);
                 return null;
             } else {
-                log.trace("Building ShareAcknowledge request to send to node {}", nodeToSend.id());
+                inFlightAcknowledgements.putAll(finalAcknowledgementsToSend);
+                if (incompleteAcknowledgements.isEmpty()) {
+                    acknowledgementsToSend.clear();
+                } else {
+                    incompleteAcknowledgements.clear();

Review Comment:
   Yes, it was not required, I have removed it now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709304100


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -84,11 +83,12 @@ public class ShareConsumeRequestManager implements RequestManager, MemberStateLi
     private Uuid memberId;
     private boolean fetchMoreRecords = false;
     private final Map<TopicIdPartition, Acknowledgements> fetchAcknowledgementsMap;
-    private final Queue<AcknowledgeRequestState> acknowledgeRequestStates;
+    private final Map<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStates;
     private final long retryBackoffMs;
     private final long retryBackoffMaxMs;
     private boolean closing = false;
     private final CompletableFuture<Void> closeFuture;
+    private boolean isAsyncDone = false;

Review Comment:
   Yes, I have made it as a local variable now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698631496


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -207,46 +212,32 @@ public void fetch(Map<TopicIdPartition, Acknowledgements> acknowledgementsMap) {
     }
 
     /**
-     * Process acknowledgeRequestStates and prepares a list of acknowledgements to be sent in the poll().
+     * Process acknowledgeRequestStates and prepares a list of acknowledgements to
+     * be sent in the poll().
      *
      * @param currentTimeMs the current time in ms.
      *
      * @return the PollResult containing zero or more acknowledgements.
      */
     private PollResult processAcknowledgements(long currentTimeMs) {
         List<UnsentRequest> unsentRequests = new ArrayList<>();
-        Iterator<AcknowledgeRequestState> iterator = acknowledgeRequestStates.iterator();
-        while (iterator.hasNext()) {
-            AcknowledgeRequestState acknowledgeRequestState = iterator.next();
-            if (acknowledgeRequestState.isProcessed()) {
-                iterator.remove();
-            } else if (!acknowledgeRequestState.maybeExpire()) {
-                if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
-                    log.trace("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
-                } else {
-                    if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
-                        acknowledgeRequestState.onSendAttempt(currentTimeMs);
-                        UnsentRequest request = acknowledgeRequestState.buildRequest(currentTimeMs);
-                        if (request != null) {
-                            unsentRequests.add(request);
-                        }
-                    }
-                }
-            } else {
-                // Fill in TimeoutException
-                for (TopicIdPartition tip : acknowledgeRequestState.acknowledgementsMap.keySet()) {
-                    metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                    acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.REQUEST_TIMED_OUT);
-                }
-                iterator.remove();
-            }
+        for (Pair<AcknowledgeRequestState> requestStates : acknowledgeRequestStates.values()) {
+            isAsyncDone = false;
+            // For commitAsync
+            maybeBuildRequest(requestStates.getFirst(), currentTimeMs, true).ifPresent(unsentRequests::add);

Review Comment:
   Yeah makes sense, I have updated now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "omkreddy (via GitHub)" <gi...@apache.org>.
omkreddy commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709350891


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -518,33 +612,48 @@ private void handleShareAcknowledgeSuccess(Node fetchTarget,
 
             if (!handler.handleResponse(response, requestVersion)) {
                 acknowledgeRequestState.onFailedAttempt(currentTimeMs);
-                if (response.error().exception() instanceof RetriableException && !closing) {
+                if (response.error().exception() instanceof RetriableException && !acknowledgeRequestState.onClose) {
                     // We retry the request until the timer expires, unless we are closing.
+                    acknowledgeRequestState.retryRequest();
                 } else {
-                    requestData.topics().forEach(topic -> topic.partitions().forEach(partition -> {
-                        TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                                partition.partitionIndex(),
-                                metadata.topicNames().get(topic.topicId()));
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                    response.data().responses().forEach(shareAcknowledgeTopicResponse -> shareAcknowledgeTopicResponse.partitions().forEach(partitionData -> {
+                        TopicIdPartition tip = new TopicIdPartition(shareAcknowledgeTopicResponse.topicId(),
+                                partitionData.partitionIndex(),
+                                metadata.topicNames().get(shareAcknowledgeTopicResponse.topicId()));
+
                         acknowledgeRequestState.handleAcknowledgeErrorCode(tip, response.error());
+                        metricsManager.recordLatency(resp.requestLatencyMs());
                     }));
-
-                    acknowledgeRequestState.processingComplete();
                 }
             } else {
-                response.data().responses().forEach(topic -> topic.partitions().forEach(partition -> {
-                    TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                            partition.partitionIndex(),
-                            metadata.topicNames().get(topic.topicId()));
-                    if (partition.errorCode() != Errors.NONE.code()) {
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                AtomicBoolean shouldRetry = new AtomicBoolean(false);

Review Comment:
   This can be primitive boolean variable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698620368


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());

Review Comment:
   Yes, have updated the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "omkreddy (via GitHub)" <gi...@apache.org>.
omkreddy commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1709165204


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -84,11 +83,12 @@ public class ShareConsumeRequestManager implements RequestManager, MemberStateLi
     private Uuid memberId;
     private boolean fetchMoreRecords = false;
     private final Map<TopicIdPartition, Acknowledgements> fetchAcknowledgementsMap;
-    private final Queue<AcknowledgeRequestState> acknowledgeRequestStates;
+    private final Map<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStates;
     private final long retryBackoffMs;
     private final long retryBackoffMaxMs;
     private boolean closing = false;
     private final CompletableFuture<Void> closeFuture;
+    private boolean isAsyncDone = false;

Review Comment:
   Can we make this as local variable?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -518,33 +609,49 @@ private void handleShareAcknowledgeSuccess(Node fetchTarget,
 
             if (!handler.handleResponse(response, requestVersion)) {
                 acknowledgeRequestState.onFailedAttempt(currentTimeMs);
-                if (response.error().exception() instanceof RetriableException && !closing) {
+                log.warn("OnFailAttempt1");

Review Comment:
   unwanted log?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -749,58 +869,104 @@ UnsentRequest buildRequest(long currentTimeMs) {
 
             BiConsumer<ClientResponse, Throwable> responseHandler = (clientResponse, error) -> {
                 if (error != null) {
-                    onFailedAttempt(currentTimeMs);
                     errorHandler.handle(nodeToSend, requestBuilder.data(), this, error, currentTimeMs);
                     processingComplete();
                 } else {
                     successHandler.handle(nodeToSend, requestBuilder.data(), this, clientResponse, currentTimeMs);
+                    if (onClose && !closeFuture.isDone()) {
+                        closeFuture.complete(null);
+                    }
                 }
             };
 
             if (requestBuilder == null) {
-                log.trace("Building ShareAcknowledge request to send to node {} failed", nodeToSend.id());
                 handleSessionErrorCode(Errors.SHARE_SESSION_NOT_FOUND);
                 return null;
             } else {
-                log.trace("Building ShareAcknowledge request to send to node {}", nodeToSend.id());
+                inFlightAcknowledgements.putAll(finalAcknowledgementsToSend);
+                if (incompleteAcknowledgements.isEmpty()) {
+                    acknowledgementsToSend.clear();
+                } else {
+                    incompleteAcknowledgements.clear();

Review Comment:
   This line may not be required. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698620806


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -378,32 +452,38 @@ public CompletableFuture<Void> acknowledgeOnClose(final Map<TopicIdPartition, Ac
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":3",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeCloseSuccess,
-                        this::handleShareAcknowledgeCloseFailure,
-                        resultHandler,
-                        true
-                ));
+
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+                // There can only be one commitSync()/close() happening at a time. So per node,

Review Comment:
   Right yeah have added a check now.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -287,22 +338,29 @@ public CompletableFuture<Map<TopicIdPartition, Acknowledgements>> commitSync(
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added sync acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added sync acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":1",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeSuccess,
-                        this::handleShareAcknowledgeFailure,
-                        resultHandler
-                ));
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // There can only be one commitSync()/close() happening at a time. So per node,

Review Comment:
   Yes makes sense, have added a check now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on PR #16727:
URL: https://github.com/apache/kafka/pull/16727#issuecomment-2260660049

   Hi @AndrewJSchofield, Thanks for the review, the formatting changes had come in due to a setting by the IDE, I have changed it now. I have addressed the remaining comments and updated the PR.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698630718


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;

Review Comment:
   Yes, had missed it, updated now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1705263060


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,6 +242,57 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState, long currentTimeMs, boolean onCommitAsync) {

Review Comment:
   Yeah so here, there are cases when if when we return `Optional.empty()`, the `isAsyncDone` variable is set to `true` and cases when we set to `false`. So as of now, we do need this variable, maybe if we have a separate map for `commitAsyncs` and `commitSyncs` in a future PR, then we can get rid of this variable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1705256472


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -565,7 +640,7 @@ private void handleShareAcknowledgeFailure(Node fetchTarget,
                 TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
                         partition.partitionIndex(),
                         metadata.topicNames().get(topic.topicId()));
-                metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getInFlightAcknowledgementsCount(tip));
                 acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.forException(error));
             }));
         } finally {

Review Comment:
   Right yeah, now I have moved all the calls into the `handleShareAcknowledge` methods for consistency.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698619373


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;

Review Comment:
   Yes, the name is quite wrong as we want to see if it is not empty actually, I have made the changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1699932061


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -876,4 +1020,34 @@ public void completeIfEmpty() {
             }
         }
     }
+
+    static class Pair<V> {
+        private V asyncRequest;
+        private V syncRequest;
+
+        public Pair(V first, V second) {

Review Comment:
   nit: The arguments to the constructor should also be `asyncRequest` and `syncRequest`



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -876,4 +1020,34 @@ public void completeIfEmpty() {
             }
         }
     }
+
+    static class Pair<V> {
+        private V asyncRequest;
+        private V syncRequest;
+
+        public Pair(V first, V second) {
+            this.asyncRequest = first;
+            this.syncRequest = second;
+        }
+
+        public void setAsyncRequest(V asyncRequest) {
+            this.asyncRequest = asyncRequest;
+        }
+
+        public void setSecond(V second) {

Review Comment:
   `setSyncRequest`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -332,22 +378,32 @@ public void commitAsync(final Map<TopicIdPartition, Acknowledgements> acknowledg
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
                         log.debug("Added async acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
                         resultCount.incrementAndGet();
+                        AcknowledgeRequestState asyncRequestState = acknowledgeRequestStates.get(nodeId).getAsyncRequest();
+                        if (asyncRequestState == null) {
+                            acknowledgeRequestStates.replace(nodeId, new Pair<>(new AcknowledgeRequestState(logContext,

Review Comment:
   There's no need to replace or construct a new Pair here. Just use `setAsyncRequest`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,6 +242,57 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState, long currentTimeMs, boolean onCommitAsync) {

Review Comment:
   Thinking about this, I would like you to remove `isAsyncDone` entirely. If this method returns `Optional.empty()`, then it means that no request was generated. Then the caller can take an appropriate action based on its return value, rather than using this piece of shared state which is more fragile.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());

Review Comment:
   I don't see this update yet.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,6 +242,57 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState, long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty() && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {

Review Comment:
   This part is wrong. `isAsyncDone` should only be set to true if `onCommitAsync` is true AND we have not built a request.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -215,37 +215,20 @@ public void fetch(Map<TopicIdPartition, Acknowledgements> acknowledgementsMap) {
      */
     private PollResult processAcknowledgements(long currentTimeMs) {
         List<UnsentRequest> unsentRequests = new ArrayList<>();
-        Iterator<AcknowledgeRequestState> iterator = acknowledgeRequestStates.iterator();
-        while (iterator.hasNext()) {
-            AcknowledgeRequestState acknowledgeRequestState = iterator.next();
-            if (acknowledgeRequestState.isProcessed()) {
-                iterator.remove();
-            } else if (!acknowledgeRequestState.maybeExpire()) {
-                if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
-                    log.trace("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
-                } else {
-                    if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
-                        acknowledgeRequestState.onSendAttempt(currentTimeMs);
-                        UnsentRequest request = acknowledgeRequestState.buildRequest(currentTimeMs);
-                        if (request != null) {
-                            unsentRequests.add(request);
-                        }
-                    }
-                }
-            } else {
-                // Fill in TimeoutException
-                for (TopicIdPartition tip : acknowledgeRequestState.acknowledgementsMap.keySet()) {
-                    metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                    acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.REQUEST_TIMED_OUT);
-                }
-                iterator.remove();
-            }
+        for (Pair<AcknowledgeRequestState> requestStates : acknowledgeRequestStates.values()) {
+            isAsyncDone = false;
+            // For commitAsync
+            maybeBuildRequest(requestStates.getAsyncRequest(), currentTimeMs, true).ifPresent(unsentRequests::add);
+
+            // Check to ensure we start processing commitSync/close only if there are no commitAsync requests left to process.
+            if (isAsyncDone)
+                maybeBuildRequest(requestStates.getSyncRequest(), currentTimeMs, false).ifPresent(unsentRequests::add);
         }
 
         PollResult pollResult = null;
         if (!unsentRequests.isEmpty()) {
             pollResult = new PollResult(unsentRequests);
-        } else if (!acknowledgeRequestStates.isEmpty()) {
+        } else if (areAnyAcknowledgementsLeft()) {

Review Comment:
   That means a side-effect of this predicate check in the else-if is to prune any remaining acknowledge request states. That seems like a significant side-effect to have which only runs conditionally. I would have make this part of this check at all.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -382,22 +438,34 @@ public CompletableFuture<Void> acknowledgeOnClose(final Map<TopicIdPartition, Ac
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":3",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeCloseSuccess,
-                        this::handleShareAcknowledgeCloseFailure,
-                        resultHandler,
-                        true
-                ));
+
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // Ensure there is no commitSync()/close() request already present as they are blocking calls
+                // and only one request can be active at a time.
+                if (acknowledgeRequestStates.get(nodeId).getSyncRequest() != null) {
+                    log.error("Attempt to call close() when there is an existing sync request for node {}", node.id());

Review Comment:
   The result needs to be considered here. Maybe in this case it's less critical than for commitSync().



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -291,18 +325,27 @@ public CompletableFuture<Map<TopicIdPartition, Acknowledgements>> commitSync(
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":1",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeSuccess,
-                        this::handleShareAcknowledgeFailure,
-                        resultHandler
-                ));
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // Ensure there is no commitSync()/close() request already present as they are blocking calls
+                // and only one request can be active at a time.
+                if (acknowledgeRequestStates.get(nodeId).getSyncRequest() != null) {
+                    log.error("Attempt to call commitSync() when there is an existing sync request for node {}", node.id());

Review Comment:
   Probably ought to have the resulting future completing exceptionally also. Otherwise, it will look like the commitSync() completed normally when it will not have been attempted and only the log line will reveal the bug.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -565,7 +640,7 @@ private void handleShareAcknowledgeFailure(Node fetchTarget,
                 TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
                         partition.partitionIndex(),
                         metadata.topicNames().get(topic.topicId()));
-                metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
+                metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getInFlightAcknowledgementsCount(tip));
                 acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.forException(error));
             }));
         } finally {

Review Comment:
   I think we ought to call `acknowledgeRequestState.onFailedAttempt` or `acknowledgeRequestState.onSuccessfulAttempt` for every single code path that handles a response to an AcknowledgeRequestState. I think that's not true at the moment but it's hard to tell because in some cases we do it in the ShareConsumeRequestManager methods, and in some cases we do it in the AcknowledgeRequestState methods. Please can you make this consistent to make it easier to validate.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -520,31 +588,38 @@ private void handleShareAcknowledgeSuccess(Node fetchTarget,
                 acknowledgeRequestState.onFailedAttempt(currentTimeMs);
                 if (response.error().exception() instanceof RetriableException && !closing) {
                     // We retry the request until the timer expires, unless we are closing.
-                } else {
-                    requestData.topics().forEach(topic -> topic.partitions().forEach(partition -> {
-                        TopicIdPartition tip = new TopicIdPartition(topic.topicId(),
-                                partition.partitionIndex(),
-                                metadata.topicNames().get(topic.topicId()));
-                        metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                        acknowledgeRequestState.handleAcknowledgeErrorCode(tip, response.error());
-                    }));
-
-                    acknowledgeRequestState.processingComplete();
+                    acknowledgeRequestState.retryRequest();

Review Comment:
   Doesn't there need to be an else clause here for the situation in which we are not retrying? That should `handleAcknowledgeErrorCode` and set the metrics I think.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -291,18 +325,27 @@ public CompletableFuture<Map<TopicIdPartition, Acknowledgements>> commitSync(
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":1",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeSuccess,
-                        this::handleShareAcknowledgeFailure,
-                        resultHandler
-                ));
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // Ensure there is no commitSync()/close() request already present as they are blocking calls
+                // and only one request can be active at a time.
+                if (acknowledgeRequestStates.get(nodeId).getSyncRequest() != null) {
+                    log.error("Attempt to call commitSync() when there is an existing sync request for node {}", node.id());
+                } else {
+                    // There can only be one commitSync()/close() happening at a time. So per node, there will be one acknowledge request state representing commitSync() and close().
+                    acknowledgeRequestStates.put(nodeId, new Pair<>(acknowledgeRequestStates.get(nodeId).getAsyncRequest(), new AcknowledgeRequestState(logContext,

Review Comment:
   There's no need to construct a new Pair here. You can use `setSyncRequest()` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1705259783


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -215,37 +215,20 @@ public void fetch(Map<TopicIdPartition, Acknowledgements> acknowledgementsMap) {
      */
     private PollResult processAcknowledgements(long currentTimeMs) {
         List<UnsentRequest> unsentRequests = new ArrayList<>();
-        Iterator<AcknowledgeRequestState> iterator = acknowledgeRequestStates.iterator();
-        while (iterator.hasNext()) {
-            AcknowledgeRequestState acknowledgeRequestState = iterator.next();
-            if (acknowledgeRequestState.isProcessed()) {
-                iterator.remove();
-            } else if (!acknowledgeRequestState.maybeExpire()) {
-                if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
-                    log.trace("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
-                } else {
-                    if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
-                        acknowledgeRequestState.onSendAttempt(currentTimeMs);
-                        UnsentRequest request = acknowledgeRequestState.buildRequest(currentTimeMs);
-                        if (request != null) {
-                            unsentRequests.add(request);
-                        }
-                    }
-                }
-            } else {
-                // Fill in TimeoutException
-                for (TopicIdPartition tip : acknowledgeRequestState.acknowledgementsMap.keySet()) {
-                    metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                    acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.REQUEST_TIMED_OUT);
-                }
-                iterator.remove();
-            }
+        for (Pair<AcknowledgeRequestState> requestStates : acknowledgeRequestStates.values()) {
+            isAsyncDone = false;
+            // For commitAsync
+            maybeBuildRequest(requestStates.getAsyncRequest(), currentTimeMs, true).ifPresent(unsentRequests::add);
+
+            // Check to ensure we start processing commitSync/close only if there are no commitAsync requests left to process.
+            if (isAsyncDone)
+                maybeBuildRequest(requestStates.getSyncRequest(), currentTimeMs, false).ifPresent(unsentRequests::add);
         }
 
         PollResult pollResult = null;
         if (!unsentRequests.isEmpty()) {
             pollResult = new PollResult(unsentRequests);
-        } else if (!acknowledgeRequestStates.isEmpty()) {
+        } else if (areAnyAcknowledgementsLeft()) {

Review Comment:
   Right yeah, we do need to prune the acknowledgements here, I have renamed the method name to make it clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1705254209


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -291,18 +325,27 @@ public CompletableFuture<Map<TopicIdPartition, Acknowledgements>> commitSync(
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":1",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeSuccess,
-                        this::handleShareAcknowledgeFailure,
-                        resultHandler
-                ));
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // Ensure there is no commitSync()/close() request already present as they are blocking calls
+                // and only one request can be active at a time.
+                if (acknowledgeRequestStates.get(nodeId).getSyncRequest() != null) {
+                    log.error("Attempt to call commitSync() when there is an existing sync request for node {}", node.id());

Review Comment:
   Right yes, have updated now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "ShivsundarR (via GitHub)" <gi...@apache.org>.
ShivsundarR commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698620074


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());
+            }
+        }
+        if (!acknowledgeRequestStates.isEmpty())
+            areAnyAcksLeft = true;
+        return areAnyAcksLeft;
+    }
+
+    private boolean isRequestStateEmpty(AcknowledgeRequestState acknowledgeRequestState) {

Review Comment:
   Yes makes sense, I have added in a method for the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #16727:
URL: https://github.com/apache/kafka/pull/16727#discussion_r1698163085


##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;

Review Comment:
   Should this line be conditional on `onCommitAsync` like the others?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -207,46 +212,32 @@ public void fetch(Map<TopicIdPartition, Acknowledgements> acknowledgementsMap) {
     }
 
     /**
-     * Process acknowledgeRequestStates and prepares a list of acknowledgements to be sent in the poll().
+     * Process acknowledgeRequestStates and prepares a list of acknowledgements to
+     * be sent in the poll().
      *
      * @param currentTimeMs the current time in ms.
      *
      * @return the PollResult containing zero or more acknowledgements.
      */
     private PollResult processAcknowledgements(long currentTimeMs) {
         List<UnsentRequest> unsentRequests = new ArrayList<>();
-        Iterator<AcknowledgeRequestState> iterator = acknowledgeRequestStates.iterator();
-        while (iterator.hasNext()) {
-            AcknowledgeRequestState acknowledgeRequestState = iterator.next();
-            if (acknowledgeRequestState.isProcessed()) {
-                iterator.remove();
-            } else if (!acknowledgeRequestState.maybeExpire()) {
-                if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
-                    log.trace("Skipping acknowledge request because previous request to {} has not been processed", acknowledgeRequestState.nodeId);
-                } else {
-                    if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
-                        acknowledgeRequestState.onSendAttempt(currentTimeMs);
-                        UnsentRequest request = acknowledgeRequestState.buildRequest(currentTimeMs);
-                        if (request != null) {
-                            unsentRequests.add(request);
-                        }
-                    }
-                }
-            } else {
-                // Fill in TimeoutException
-                for (TopicIdPartition tip : acknowledgeRequestState.acknowledgementsMap.keySet()) {
-                    metricsManager.recordFailedAcknowledgements(acknowledgeRequestState.getAcknowledgementsCount(tip));
-                    acknowledgeRequestState.handleAcknowledgeErrorCode(tip, Errors.REQUEST_TIMED_OUT);
-                }
-                iterator.remove();
-            }
+        for (Pair<AcknowledgeRequestState> requestStates : acknowledgeRequestStates.values()) {
+            isAsyncDone = false;
+            // For commitAsync
+            maybeBuildRequest(requestStates.getFirst(), currentTimeMs, true).ifPresent(unsentRequests::add);

Review Comment:
   I think the getFirst()/getSecond() could be better named getAsyncRequest()/getSyncRequest() since that's what first and second actually mean.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -378,32 +452,38 @@ public CompletableFuture<Void> acknowledgeOnClose(final Map<TopicIdPartition, Ac
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":3",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeCloseSuccess,
-                        this::handleShareAcknowledgeCloseFailure,
-                        resultHandler,
-                        true
-                ));
+
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+                // There can only be one commitSync()/close() happening at a time. So per node,

Review Comment:
   This is true but you could ensure that it's true in the code also.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;

Review Comment:
   This confuses me. It says "if there are no async or sync acks", but then it sets `areAnyAcksLeft = true`. Seems that the logic and the names of the methods could be a bit clearer.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());

Review Comment:
   Shouldn't this be `iterator.remove()`?



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -378,32 +452,38 @@ public CompletableFuture<Void> acknowledgeOnClose(final Map<TopicIdPartition, Ac
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added closing acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":3",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeCloseSuccess,
-                        this::handleShareAcknowledgeCloseFailure,
-                        resultHandler,
-                        true
-                ));
+
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+                // There can only be one commitSync()/close() happening at a time. So per node,
+                // there will be one acknowledge request state.
+                acknowledgeRequestStates.put(nodeId, new Pair<>(acknowledgeRequestStates.get(nodeId).getFirst(),
+                        new AcknowledgeRequestState(logContext,
+                                ShareConsumeRequestManager.class.getSimpleName() + ":1",

Review Comment:
   The name with `:3` at the end has been lost with this PR. Please restore this.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -287,22 +338,29 @@ public CompletableFuture<Map<TopicIdPartition, Acknowledgements>> commitSync(
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added sync acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added sync acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
                     }
                 }
-                acknowledgeRequestStates.add(new AcknowledgeRequestState(logContext,
-                        ShareConsumeRequestManager.class.getSimpleName() + ":1",
-                        deadlineMs,
-                        retryBackoffMs,
-                        retryBackoffMaxMs,
-                        sessionHandler,
-                        nodeId,
-                        acknowledgementsMapForNode,
-                        this::handleShareAcknowledgeSuccess,
-                        this::handleShareAcknowledgeFailure,
-                        resultHandler
-                ));
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
+                // There can only be one commitSync()/close() happening at a time. So per node,

Review Comment:
   This is true but it would be simple to ensure that using a bit of defensive code. If we made an error in the code later, we might end up overwriting something we didn't expect and not notice.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -324,44 +382,60 @@ public void commitAsync(final Map<TopicIdPartition, Acknowledgements> acknowledg
             Node node = cluster.nodeById(nodeId);
             if (node != null) {
                 Map<TopicIdPartition, Acknowledgements> acknowledgementsMapForNode = new HashMap<>();
+
+                acknowledgeRequestStates.putIfAbsent(nodeId, new Pair<>(null, null));
+
                 for (TopicIdPartition tip : sessionHandler.sessionPartitions()) {
                     Acknowledgements acknowledgements = acknowledgementsMap.get(tip);
                     if (acknowledgements != null) {
                         acknowledgementsMapForNode.put(tip, acknowledgements);
 
                         metricsManager.recordAcknowledgementSent(acknowledgements.size());
-                        log.debug("Added async acknowledge request for partition {} to node {}", tip.topicPartition(), node.id());
+                        log.debug("Added async acknowledge request for partition {} to node {}", tip.topicPartition(),
+                                node.id());
                         resultCount.incrementAndGet();
+                        if (acknowledgeRequestStates.get(nodeId).getFirst() == null) {

Review Comment:
   You should assign the result of `acknowledgeRequestStates.get(nodeId).getFirst()` to a variable so that you can avoid repeatedly evaluating this expression. I think you're doing it 3 times when once would be enough and also make the code clearer.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());
+            }
+        }
+        if (!acknowledgeRequestStates.isEmpty())
+            areAnyAcksLeft = true;
+        return areAnyAcksLeft;
+    }
+
+    private boolean isRequestStateEmpty(AcknowledgeRequestState acknowledgeRequestState) {

Review Comment:
   This could be clearer. I think `ars == null && ars.acksToSend.isEmpty && ars.ia.isEmpty() && ars.ifa.isEmpty()`.



##########
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ShareConsumeRequestManager.java:
##########
@@ -259,11 +250,71 @@ private PollResult processAcknowledgements(long currentTimeMs) {
         return pollResult;
     }
 
+    private Optional<UnsentRequest> maybeBuildRequest(AcknowledgeRequestState acknowledgeRequestState,
+            long currentTimeMs, boolean onCommitAsync) {
+        if (acknowledgeRequestState == null || (!closing && acknowledgeRequestState.acknowledgementsToSend.isEmpty()
+                && acknowledgeRequestState.incompleteAcknowledgements.isEmpty())) {
+            if (onCommitAsync) {
+                isAsyncDone = true;
+            }
+            return Optional.empty();
+        } else if (!acknowledgeRequestState.maybeExpire()) {
+            if (nodesWithPendingRequests.contains(acknowledgeRequestState.nodeId)) {
+                log.debug("Skipping acknowledge request because previous request to {} has not been processed",
+                        acknowledgeRequestState.nodeId);
+            } else {
+                if (acknowledgeRequestState.canSendRequest(currentTimeMs)) {
+                    acknowledgeRequestState.onSendAttempt(currentTimeMs);
+                    if (onCommitAsync) {
+                        isAsyncDone = true;
+                    }
+                    return Optional.of(acknowledgeRequestState.buildRequest(currentTimeMs));
+                }
+            }
+        } else {
+            // Fill in TimeoutException
+            for (TopicIdPartition tip : acknowledgeRequestState.incompleteAcknowledgements.keySet()) {
+                metricsManager
+                        .recordFailedAcknowledgements(acknowledgeRequestState.getIncompleteAcknowledgementsCount(tip));
+                acknowledgeRequestState.handleAcknowledgeTimedOut(tip);
+            }
+            acknowledgeRequestState.incompleteAcknowledgements.clear();
+            isAsyncDone = true;
+        }
+        return Optional.empty();
+    }
+
+    private boolean areAnyAcknowledgementsLeft() {
+        boolean areAnyAcksLeft = false;
+        Iterator<Map.Entry<Integer, Pair<AcknowledgeRequestState>>> iterator = acknowledgeRequestStates.entrySet()
+                .iterator();
+        while (iterator.hasNext()) {
+            Map.Entry<Integer, Pair<AcknowledgeRequestState>> acknowledgeRequestStatePair = iterator.next();
+            if (isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getFirst())
+                    && isRequestStateEmpty(acknowledgeRequestStatePair.getValue().getSecond())) {
+                areAnyAcksLeft = true;
+            } else if (!closing) {
+                acknowledgeRequestStates.remove(acknowledgeRequestStatePair.getKey());
+            }
+        }
+        if (!acknowledgeRequestStates.isEmpty())
+            areAnyAcksLeft = true;
+        return areAnyAcksLeft;
+    }
+
+    private boolean isRequestStateEmpty(AcknowledgeRequestState acknowledgeRequestState) {

Review Comment:
   You could even make a method `boolean AcknowledgeRequestState.isEmpty()` so that this code was not intimately aware of the three maps within.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager [kafka]

Posted by "omkreddy (via GitHub)" <gi...@apache.org>.
omkreddy merged PR #16727:
URL: https://github.com/apache/kafka/pull/16727


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org