You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "adixitconfluent (via GitHub)" <gi...@apache.org> on 2024/10/18 08:55:10 UTC

[PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

adixitconfluent opened a new pull request, #17539:
URL: https://github.com/apache/kafka/pull/17539

   ### About
   `minBytes` is also a constraint that should be used to delay ShareFetch requests.
   
   ### Testing
   The added code has been tested with the help of unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816818571


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hi @junrao , I understand your point now. Here's my proposed solution changes- 
   1. Remove update of `latestFetchOffsetMetadata` via `acquire` method
   2. In `onComplete`, after I complete the call of `ShareFetchUtils.processFetchResponse`(which internally completes the call of `acquire` method), I will do a `readFromLog` for `topicPartitionData` with their latest fetch offset and update the share partition's `latestFetchOffsetMetadata`. Note this call to `readFromLog` will always happen despite the `tryComplete` `readFromLog` happens or not.
   Please let me know what you think of this approach.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1818158187


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;

Review Comment:
   The explanation is present in this conversation thread https://github.com/apache/kafka/pull/17539#discussion_r1815531648



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806720826


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,10 +133,13 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
+        if (!topicPartitionData.isEmpty()) {
+            replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, false);
+            if (!replicaManagerFetchDataFromTryComplete.isEmpty())
+                return forceComplete();
+        }
         log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +

Review Comment:
   You're right... I'll change it to `
   log.info("Fetch cannot be completed for the partitions in the share fetch request for group {}, member {}, " +
                   "topic partitions {}", shareFetchData.groupId(),
                   shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet())
   `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826561286


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();

Review Comment:
   Hi @junrao , we can do that but then once we are in the `else` of this check -` if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData))`, after we have release the partitions lock, we also need to do `partitionsToComplete.clear()`. Hence, I've tried to avoid doing this by  assigning `partitionsToComplete` once we are sure that we can do a `forceComplete()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1809422393


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);
             shareFetchData.future().complete(result);
         } catch (Exception e) {
             log.error("Error processing delayed share fetch request", e);
             shareFetchData.future().completeExceptionally(e);
         } finally {
             // Releasing the lock to move ahead with the next request in queue.
-            releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            releasePartitionLocks(shareFetchData.groupId(), fetchResponseData.keySet());

Review Comment:
   Reading `readFromLog` functionality, I don't see an indication where it doesn't return all the partitions sent to it, however I'll still make the change to use `topicPartitionData` to be on the safe side.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828853882


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",

Review Comment:
   my bad, you're right about both.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816512680


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   mkaes sense, IIUC, whenever there is an acknowledge or acquisition lock timeout or release records on session close, I should update `latestFetchOffsetMetadata` in `SharePartition` to `Optional.empty()`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1810810655


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +197,62 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        boolean minBytesSatisfied = false;
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(

Review Comment:
   Hi @junrao , agreed this is a much better approach. Just one clarification - 
   > This requires us to maintain the file position for SharePartition.endOffset
   
   You mean the file position of the latest offset that was fetched for the share partition, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1812747750


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1475,6 +1486,10 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    LogOffsetMetadata latestFetchOffsetMetadata() {

Review Comment:
   I've moved it to protected. It cannot be private since it is utilized in `DelayedShareFetch`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826157479


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1576,6 +1582,34 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateEndOffsetAndFetchOffsetMetadata(long endOffset, Optional<LogOffsetMetadata> fetchOffsetMetadata) {

Review Comment:
   There is no need to pass in `fetchOffsetMetadata` since it's always empty.
   
   updateEndOffsetAndFetchOffsetMetadata => updateEndOffsetAndResetFetchOffsetMetadata?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   This problem is still there?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLogAndUpdateFetchOffsetMetadata(topicPartitionData);
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    if (!replicaManagerReadResponse.isEmpty())
+                        partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Should we reset `partitionsToComplete` and `partitionsAlreadyFetched` too when we release the locks?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1576,6 +1582,34 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateEndOffsetAndFetchOffsetMetadata(long endOffset, Optional<LogOffsetMetadata> fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            this.endOffset = endOffset;

Review Comment:
   Let's be consistent with the usage of `this`. Most other places don't use `this`. 



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1576,6 +1582,34 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateEndOffsetAndFetchOffsetMetadata(long endOffset, Optional<LogOffsetMetadata> fetchOffsetMetadata) {
+        lock.writeLock().lock();

Review Comment:
   All callers hold the lock. So we could remove the locking here and add a comment that the caller is expected to hold the lock when calling this method.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLogAndUpdateFetchOffsetMetadata(topicPartitionData);
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    if (!replicaManagerReadResponse.isEmpty())

Review Comment:
   This check is unnecessary since partitionsAlreadyFetched initializes to empty.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +239,159 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLogAndUpdateFetchOffsetMetadata(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(replicaManagerReadResponseData);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadataForMissingTopicPartitions(

Review Comment:
   updateFetchOffsetMetadataForMissingTopicPartitions => updateFetchOffsetMetadata ?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +239,159 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLogAndUpdateFetchOffsetMetadata(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }

Review Comment:
   This code can be a bit simpler.
   
   ```
   topicPartitionData.forEach((topicIdPartition, partitionData) -> {
       SharePartition sharePartition = sharePartitions.get(topicIdPartition);
       if (sharePartition.fetchOffsetMetadata().isEmpty()) {
           missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
       }
   });
   ```



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();

Review Comment:
   Should we just assign the return value to partitionsToComplete directly? We already acquired the locks for those partitions and partitionsToComplete is the only place to track them for releasing.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +239,159 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLogAndUpdateFetchOffsetMetadata(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(replicaManagerReadResponseData);

Review Comment:
   Perhaps do this in the caller? Then the purpose of the method is simpler and the method name can just be `maybeReadFromLog`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2450647031

   > @adixitconfluent : Thanks for the updated PR. A few more comments. Also, it seems the previous comments on SharePartition haven't been addressed?
   
   Hi @junrao, I am still in the middle of the refactor. I will raise a re-review once I am done. I was just resolving the comments for which I pushed the fixes in my recent commits, probably that gave a false idea that I was done addressing your changes, sorry about that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816800887


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            TopicIdPartition topicIdPartition = tpLogResult._1();
+            LogReadResult logResult = tpLogResult._2();
+            FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+            responseData.put(topicIdPartition, new FetchPartitionOffsetData(fetchPartitionData, logResult.info().fetchOffsetMetadata));

Review Comment:
   I can confirm this will return information about 0th offset.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828758874


##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,125 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testDelayedShareFetchTryCompleteReturnsFalseDueToMinBytesOnFirstSharePartitionFetch() {
+        String groupId = "grp";
+        Uuid topicId = Uuid.randomUuid();
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(topicId, new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(topicId, new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        when(sp0.maybeAcquireFetchLock()).thenReturn(true);
+        when(sp1.maybeAcquireFetchLock()).thenReturn(true);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                2, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        when(sp0.canAcquireRecords()).thenReturn(true);
+        when(sp1.canAcquireRecords()).thenReturn(false);
+        when(sp0.acquire(any(), anyInt(), any())).thenReturn(
+            ShareAcquiredRecords.fromAcquiredRecords(new ShareFetchResponseData.AcquiredRecords().setFirstOffset(0).setLastOffset(3).setDeliveryCount((short) 1)));
+
+        // We are testing the case when the share partition is getting fetched for the first time, hence we are using 1
+        // as the file position, so it doesn't satisfy the minBytes(2).
+        when(sp0.fetchOffsetMetadata()).thenReturn(Optional.of(new LogOffsetMetadata(0, 1, 0)));
+        LogOffsetMetadata hwmOffsetMetadata = new LogOffsetMetadata(1, 1, 1);
+        mockTopicIdPartitionFetchBytes(replicaManager, tp0, hwmOffsetMetadata);
+
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
+            .withShareFetchData(shareFetchData)
+            .withSharePartitions(sharePartitions)
+            .withReplicaManager(replicaManager)
+            .build());
+        assertFalse(delayedShareFetch.isCompleted());
+
+        // Since sp1 can be acquired, tryComplete should return true.

Review Comment:
   my bad, I've corrected it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828824435


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();

Review Comment:
   Yes, there have been comments above where I left some variables as null, and it was pointed out that I need to initialize them to avoid null checks at different places, then we just need to do empty checks.. Hence, I've implemented it in this manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828860825


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);

Review Comment:
   you're right, I've changed the line to `releasePartitionLocks(topicPartitionData.keySet())`. If the `forceComplete` is successful, then the partition locks are released from `onComplete` finally block and it doesn't concern here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1807312461


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;

Review Comment:
   I can get rid of it, but then the variable name `replicaManagerFetchDataFromTryComplete` won't make sense if it is getting some values in `onComplete`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806619158


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());

Review Comment:
   Hi @apoorvmittal10 , IIUC, minBytes is utilized in `replicaManager.fetchMessages` functionality not in `replicaManager.readFromLog` [here](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1729). The way it calculates the `accumulatedBytes` is the same way I have done it in my code ([original code reference](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1718)). I don't see the usage of `params.minBytes` in `readFromLog` functionality



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1809669520


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +197,62 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        boolean minBytesSatisfied = false;
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(

Review Comment:
   1. This approach is ok, but probably not the most efficient. `replicaManager.readFromLog` is relatively expensive. To avoid calling it on every HWM change, the DelayedFetch maintains the file position of the fetch offset and compares it with the file position of HWM to estimate the fetchable bytes. We could potentially do the same thing here. This requires us to maintain the file position for SharePartition.endOffset.
   2. Ideally, we need to take into account the size of those non-acquirable batches in SharePartition when estimating the fetchable bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1810373760


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +197,62 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        boolean minBytesSatisfied = false;
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(

Review Comment:
   Thanks a lot for suggestion @junrao. This is good. 
   
   I have aquestion on 2.
   > Ideally, we need to take into account the size of those non-acquirable batches in SharePartition when estimating the fetchable bytes.
   
   Though it's an ideal solution but the next fetch offset being prior to endOffset should be rare i.e. when some records are released or timedout. So I think we can avoid calculating non-acquirable and proceed to fetch anyways if our criteria from end offset to HWM meets. We can have the min bytes check after the fetch as currently in the PR. Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2427455305

   Hi @junrao @apoorvmittal10 , please review my PR when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815592778


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   `updateLatestFetchOffsetMetadata()` is only called from `tryComplete` when we find that there is a share partition whose offset metadata is not present (because in that case only, we call `readFromLog`). However, for any other case, we can only update the `latestFetchOffsetMetadata` via `acquire` method only. Else we'll have to call `readFromLog` from `tryComplete` in all scenarios to directly update share partition's `latestFetchOffsetMetadata`. Anything wrong in that understanding?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816512680


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   mkaes sense, IIUC, whenever there is an acknowledge or acquisition lock timeout or release records on session close, I'll `latestFetchOffsetMetadata` in `SharePartition` to `Optional.empty()` in my latest commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816818571


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hi @junrao , I understand your point now. Here's my proposed solution changes- 
   1. Remove update of `latestFetchOffsetMetadata` via `acquire` method
   2. In `onComplete`, after I complete the call of `ShareFetchUtils.processFetchResponse`(which internally completes the call of `acquire` method), I will do a `readFromLog` for `topicPartitionData` with their latest fetch offset and update the share partition's `latestFetchOffsetMetadata`. Note this call to `readFromLog` will always happen despite the `tryComplete` `readFromLog` happens or not.
   
   Please let me know what you think of this approach.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816827231


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hi @junrao, understood, so if there is any calls to acknowledge/acquisition lock timeout/release acquired records on session close, I should update the latestFetchOffsetMetadata to `Optional.empty()`. Then when the next tryComplete call comes, it will update the `latestFetchOffsetMetadata` and we will have the most recent result. Or is my understanding incorrect?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1807413812


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }
+        }
+        // This call is coming from onComplete, hence we return the response data irrespective of whether minBytes is
+        // satisfied or not.
+        if (hasRequestTimedOut)
+            return responseData;

Review Comment:
   Hi @apoorvmittal10, I went through the code for onComplete of `DelayedFetch` ([code reference](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DelayedFetch.scala#L176)), it seems that it returns whatever it gets from `replicaManager.readFromLog`, so the current code is correct in that case. Please let me know if I am wrong. cc - @junrao 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1818157944


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;

Review Comment:
   Yes, it is handled here https://github.com/apache/kafka/pull/17539#discussion_r1816519200



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826092331


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions == null || missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new FetchOffsetMetadataUpdateResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(missingFetchOffsetMetadataTopicPartitions, replicaManagerReadResponseData);
+    }
+
+    private FetchOffsetMetadataUpdateResult updateFetchOffsetMetadataForMissingTopicPartitions(
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions,
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.getOrDefault(topicIdPartition, null);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new FetchOffsetMetadataUpdateResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   Hi @junrao, I have changed the exception handling to a top level exception handling in `tryComplete` to combat with this scenario.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1829020183


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   hey @apoorvmittal10 , we are doing partition level error handling in `onComplete` using `sharePartitionManager.handleFetchException`. When we get an exception in this line `replicaManager.getPartitionOrException`, we directly call `forceComplete` (mentioned in above comments), and that does a partition level handling.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830395752


##########
core/src/main/java/kafka/server/share/ShareFetchUtils.java:
##########
@@ -64,11 +64,7 @@ static Map<TopicIdPartition, ShareFetchResponseData.PartitionData> processFetchR
             TopicIdPartition topicIdPartition = entry.getKey();
             FetchPartitionData fetchPartitionData = entry.getValue();
 
-            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
-                continue;
-            }
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);

Review Comment:
   yes, I think changing the log line to below makes more sense.
   ```
   log.trace("Acquired records: {} for topicIdPartition: {}", shareAcquiredRecords, topicIdPartition);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1829235324


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();

Review Comment:
   If the concern is with additional null check then I would recommend general helper methods. My concern is with creating additional maps when they are always re-referenced.
   
   ```
       private <K,V> void addToNullableMap(Map<K, V> map, K key, V value) {
           if (map == null) {
               map = new LinkedHashMap<>();
           }
           map.put(key, value);
       }
   
       private boolean isMapEmpty(Map<?,?> map) {
           return map == null || map.isEmpty();
       }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1829240438


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   It's missing yet, you can log a jira for me to fix as I am doing handling anyways.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1811067709


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +197,62 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        boolean minBytesSatisfied = false;
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(

Review Comment:
   > Though it's an ideal solution but the next fetch offset being prior to endOffset should be rare i.e. when some records are released or timedout. So I think we can avoid calculating non-acquirable and proceed to fetch anyways if our criteria from end offset to HWM meets. We can have the min bytes check after the fetch as currently in the PR. Wdyt?
   
   Yes, this sounds reasonable. We can just ignore the non-acquirable records for now.
   
   > You mean the file position of the latest offset that was fetched for the share partition, right?
   
   Basically, we need to maintain endOffset as a `LogOffsetMetadata`, which contains segment position. We can then use `LogOffsetMetadata.positionDiff` to calculate the available bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816703702


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            TopicIdPartition topicIdPartition = tpLogResult._1();
+            LogReadResult logResult = tpLogResult._2();
+            FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+            responseData.put(topicIdPartition, new FetchPartitionOffsetData(fetchPartitionData, logResult.info().fetchOffsetMetadata));

Review Comment:
   So if we are fetching from offset 0 and gets response from log for 0-1000 offsets then `logResult.info().fetchOffsetMetadata` contains information for 0 offset or 1000th offset i.e. which offset metadata does it hold?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1813657687


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -89,25 +91,45 @@ public void onComplete() {
             return;
 
         Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
-        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
-        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
+        if (topicPartitionDataFromTryComplete.isEmpty())
             topicPartitionData = acquirablePartitions();
-            if (topicPartitionData.isEmpty()) {
-                // No locks for share partitions could be acquired, so we complete the request with an empty response.
-                shareFetchData.future().complete(Collections.emptyMap());
-                return;
-            }
-            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
-        } else {
-            // tryComplete invoked forceComplete, so we can use the topic partitions data and replica manager response data from tryComplete.
+        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
+        else
             topicPartitionData = topicPartitionDataFromTryComplete;
-            fetchResponseData = replicaManagerFetchDataFromTryComplete;
+
+        if (topicPartitionData.isEmpty()) {
+            // No locks for share partitions could be acquired, so we complete the request with an empty response.
+            shareFetchData.future().complete(Collections.emptyMap());
+            return;
         }
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
+                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+            Map<TopicIdPartition, LogOffsetMetadata> fetchOffsetMetadata = new HashMap<>();
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                fetchOffsetMetadata.put(topicIdPartition, logResult.info().fetchOffsetMetadata);
+                return BoxedUnit.UNIT;
+            });
+
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, fetchOffsetMetadata, sharePartitionManager, replicaManager);

Review Comment:
   It's kind of late to set the offset metadata here since we will acquire the fetch records and move the next fetch offset. This means the cached offset metadata doesn't match the next fetch offset.
   
   I was thinking of doing the following. In `tryComplete`, if the offset metadata doesn't exist, we call `replicaManager.readFromLog` to populate the offset metadata. We can then proceed with the minByte estimation. If SharePartition.endOffset moves, we invalidate the offset metadata.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -197,58 +219,25 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    /**
-     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
-     */
-    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
-                                                                      boolean hasRequestTimedOut) {
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
-            shareFetchData.groupId(), shareFetchData.fetchParams());
-        boolean minBytesSatisfied = false;
-        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-        try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            AtomicLong accumulatedBytes = new AtomicLong(0);
-
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
-                return BoxedUnit.UNIT;
-            });
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
-
-            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
-                minBytesSatisfied = true;
-        } catch (Exception e) {
-            log.error("Error processing delayed share fetch request", e);
-        } finally {
-            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
-            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
-            // tryComplete/onComplete call complete successfully.
-            if (!minBytesSatisfied && !hasRequestTimedOut) {
-                // Releasing the lock to move ahead with the next request in queue.
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+    boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        AtomicLong accumulatedSize = new AtomicLong(0);
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+            LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+            LogOffsetMetadata endOffsetMetadata = offsetSnapshot.highWatermark;

Review Comment:
   We need to check `shareFetchData.fetchParams().isolation` to decide whether to use HWM or lastStableOffset.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -197,58 +219,25 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    /**
-     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
-     */
-    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
-                                                                      boolean hasRequestTimedOut) {
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
-            shareFetchData.groupId(), shareFetchData.fetchParams());
-        boolean minBytesSatisfied = false;
-        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-        try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            AtomicLong accumulatedBytes = new AtomicLong(0);
-
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
-                return BoxedUnit.UNIT;
-            });
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
-
-            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
-                minBytesSatisfied = true;
-        } catch (Exception e) {
-            log.error("Error processing delayed share fetch request", e);
-        } finally {
-            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
-            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
-            // tryComplete/onComplete call complete successfully.
-            if (!minBytesSatisfied && !hasRequestTimedOut) {
-                // Releasing the lock to move ahead with the next request in queue.
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+    boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        AtomicLong accumulatedSize = new AtomicLong(0);
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+            LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+            LogOffsetMetadata endOffsetMetadata = offsetSnapshot.highWatermark;
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                return;
             }
-        }
-        // We can return the replica manager fetch response for the following 2 cases -
-        // 1. This call is coming from onComplete, hence we return the response data irrespective of whether minBytes is
-        // satisfied or not.
-        // 2. Return the response if minBytes criteria is satisfied (request is coming from tryComplete).
-        if (hasRequestTimedOut || minBytesSatisfied)
-            return responseData;
-        // Return an empty map if replica manager fetch does not satisfy minBytes (request is coming from tryComplete).
-        return Collections.emptyMap();
+
+            // we take the partition fetch size as upper bound when accumulating the bytes (skip if a throttled partition)
+            long bytesAvailable = sharePartition.latestFetchOffsetMetadata() == null ?
+                Math.min(endOffsetMetadata.relativePositionInSegment, partitionData.maxBytes) :
+                Math.min(endOffsetMetadata.positionDiff(sharePartition.latestFetchOffsetMetadata()), partitionData.maxBytes);

Review Comment:
   We need to check if the two offset metadata are on the same segment first before using `positionDiff`. Also, we need to handle the case when HWM doesn't have offset metadata. We can just follow the logic in `DelayedFetch`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806669871


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }
+        }
+        // This call is coming from onComplete, hence we return the response data irrespective of whether minBytes is
+        // satisfied or not.
+        if (hasRequestTimedOut)
+            return responseData;

Review Comment:
   Hmmm, is this same for regular fetch operations as well?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }

Review Comment:
   And why don't we want to release the partition locks from onComplete?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);

Review Comment:
   fetchResponseData can still be empty, though processFetchResponse handles the empty check, is it intended? Though no harm, just checking with you.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();

Review Comment:
   In most scenarios the request might have minBytes, hence do you always want to initialize a hash map? Mostly it will be overriden with `responseData` map. So can't it be null? Moreover can't it be simpy a boolean variable i.e. 
   
   boolean minBytesSatisfied = false
   
   if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
                   replicaManagerFetchSatisfyingMinBytes = responseData;
   =>
   if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
                   minBytesSatisfied = true;
   
   if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) { 
   =>
   if (!minBytesSatisfied && !hasRequestTimedOut) {
   
   return replicaManagerFetchSatisfyingMinBytes;
   =>
   return Collections.emptyMap()
   



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,10 +133,13 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
+        if (!topicPartitionData.isEmpty()) {
+            replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, false);
+            if (!replicaManagerFetchDataFromTryComplete.isEmpty())
+                return forceComplete();
+        }
         log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +

Review Comment:
   Now we can come to this code path getting no result from `replicaManagerFetchData`. Hence is the log line still correct?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;

Review Comment:
   Do you need this extra variable or can just write later, if needed? And then no need of else block below.
   
   replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, true);



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);
             shareFetchData.future().complete(result);
         } catch (Exception e) {
             log.error("Error processing delayed share fetch request", e);
             shareFetchData.future().completeExceptionally(e);
         } finally {
             // Releasing the lock to move ahead with the next request in queue.
-            releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            releasePartitionLocks(shareFetchData.groupId(), fetchResponseData.keySet());

Review Comment:
   What about if partitions were locked but no response in data aarived then will the lock be correctly released?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806619158


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());

Review Comment:
   Hi @apoorvmittal10 , IIUC, minBytes is utilized in `replicaManager.fetchMessages` functionality [here](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1729) not in `replicaManager.readFromLog`. The way it calculates the `accumulatedBytes` is the same way I have done it in my code ([original code reference](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1718)). I don't see the usage of `params.minBytes` in `readFromLog` functionality



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828532884


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();

Review Comment:
   We initialize the variable in constructor then re-assign while creating another LinkedHashMap in acquirablePartitions() method. Are we are initializing `partitionsToComplete` here to save null check? Can't we re-use already initialized LinkedHashMap()?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +

Review Comment:
   There are 2 checks in the if condition (anyTopicIdPartitionHasLogReadError and isMinBytesSatisfied) but the log says that minBytes criteria is not satified. I this correct log statement?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(

Review Comment:
   Shouldn't the name be `maybeUpdateFetchOffsetMetadata` as it depends on the log read result?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {

Review Comment:
   So did we not chose to implement https://github.com/apache/kafka/pull/17539/files#r1816530978 rather initialize with LinkedHashMap which will hardly be filled?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -169,9 +205,9 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = new LinkedHashMap<>();
 
         shareFetchData.partitionMaxBytes().keySet().forEach(topicIdPartition -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
             if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
                 return;
             }

Review Comment:
   Same elsewhere.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -169,9 +205,9 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = new LinkedHashMap<>();
 
         shareFetchData.partitionMaxBytes().keySet().forEach(topicIdPartition -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
             if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
                 return;
             }

Review Comment:
   We have now passed `sharePartitions` map which contains topicIdPartition and sharePartition itself then why are we iterating on `shareFetchData.partitionMaxBytes().keySet()` and doing a null check, why not to iterate in `sharePartitions` map itself? Also make the map of `sharePartitions` in SPM as of type LinkedHashMap then.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();
+        this.partitionsAlreadyFetched = new LinkedHashMap<>();

Review Comment:
   Again we reset `partitionsAlreadyFetched` to response from `replicaManagerReadResponse`. Why to have such instances created when anyways we have to re-assign?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;

Review Comment:
   We are re-assigning the already initalized variables in constructor. I would say we should have null check handling in `onComplete` rather creating resources which never gets utilized.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);

Review Comment:
   Seems an incorrect error handling of release acquired partitions to me. Say line 155 acquires partitions and `topicPartitionData` is set. And we get an exception in `isMinBytesSatisfied` method (which anyways call getPartitionOrException method) or elsewhere then the locks released at line 193 will not release any locks as they are invoked on `partitionsToComplete` which is not yet set. Moreover if forceComplete call is successful then the acquired partitions will anyways not be released.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it other FetchIsolation types.

Review Comment:
   ```suggestion
           // extend it to support other FetchIsolation types.
   ```



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",

Review Comment:
   Isn't the log incorrect as it says the the log does not contain topicIdPartition rather the response exists but it errored. Also do you need to log `replicaManagerLogReadResult` or complete `replicaManagerReadResponseData`? Shouldn't we be logging former which corresponds to topic id partition?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,34 +113,27 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There can't be a case when we have a non-empty logReadResponse, and we have a fresh topicPartitionData
+                // using tryComplete because purgatory ensures that 2 tryCompletes calls do not happen at the same time.

Review Comment:
   I understand there can't be concurrent calls to tryComplete but I didn't get this comment. Though forceComplete() cannot be called twice. But forceComplete() on expiration can be on different thread and tryComplete() on different can this change any behaviour here?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {

Review Comment:
   nit: I personally find the code hard to read with nested if/else blocks, same is the case here. Though I leave it on you.
   
   ```
   if (topicPartitionData.isEmpty()) {
        log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
                           "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
                       shareFetchData.partitionMaxBytes().keySet());
       return false;
   }
   
   // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
   // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
   // those topic partitions.
   Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
   if (!anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) && !isMinBytesSatisfied(topicPartitionData)) {
          log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
                               "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
          shareFetchData.partitionMaxBytes().keySet());
          releasePartitionLocks(topicPartitionData.keySet());
          return false;
   }
   
   partitionsToComplete = topicPartitionData;
   partitionsAlreadyFetched = replicaManagerReadResponse;
   ....
   ....
   



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   What @junrao meant was that it should not be a top level rather partition level error.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {

Review Comment:
   nit: Will `forEach` be more convenient here then you don't need to call entry.getKey and entry.getValue?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,34 +113,27 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There can't be a case when we have a non-empty logReadResponse, and we have a fresh topicPartitionData
+                // using tryComplete because purgatory ensures that 2 tryCompletes calls do not happen at the same time.
+                responseData = combineLogReadResponse(topicPartitionData);
+
+            Map<TopicIdPartition, FetchPartitionData> fetchPartitionsData = new LinkedHashMap<>();
+            for (Map.Entry<TopicIdPartition, LogReadResult> entry : responseData.entrySet())
+                fetchPartitionsData.put(entry.getKey(), entry.getValue().toFetchPartitionData(false));
+
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchPartitionsData, sharePartitionManager, replicaManager);
             shareFetchData.future().complete(result);

Review Comment:
   Can be merged together.
   
   ```
    shareFetchData.future().complete(ShareFetchUtils.processFetchResponse(shareFetchData, fetchPartitionsData, sharePartitionManager, replicaManager));



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   Shouldn't we iterate on `sharePartitions` map passed in delayed share fetch which will guarantee that SharePartition cannot be null it can only be fenced (fenced handling has been separate), hence no null check is required. Also no `anySharePartitionNoLongerExists()` method call is required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828579315


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,34 +113,27 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There can't be a case when we have a non-empty logReadResponse, and we have a fresh topicPartitionData
+                // using tryComplete because purgatory ensures that 2 tryCompletes calls do not happen at the same time.

Review Comment:
   I understand there can't be concurrent calls to tryComplete but I didn't get this comment. Though forceComplete() cannot be called twice. But forceComplete() on expiration can be on different thread and tryComplete() on different (that's my understanding as per DelayedOperation code I have seen) can this change any behaviour here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828831006


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {

Review Comment:
   I have been asked to add else blocks in the above comments on this PR, hence I don' think I should change it again. https://github.com/apache/kafka/pull/17539#discussion_r1819610603



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1829237621


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {

Review Comment:
   I leave it on @junrao to decide then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830199934


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -51,21 +55,26 @@ public class DelayedShareFetch extends DelayedOperation {
 
     private static final Logger log = LoggerFactory.getLogger(DelayedShareFetch.class);
 
-    private final ShareFetchData shareFetchData;
+    private final ShareFetchData partitionsToComplete;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsAcquired;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
-            ShareFetchData shareFetchData,
+            ShareFetchData partitionsToComplete,

Review Comment:
   This is actually a super set of the partitions to complete. So, it's better to just keep the name shareFetchData. It would be useful to add a comment that the partitions to be completed are given by sharePartitions and is a subset of shareFetchData.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -51,21 +55,26 @@ public class DelayedShareFetch extends DelayedOperation {
 
     private static final Logger log = LoggerFactory.getLogger(DelayedShareFetch.class);
 
-    private final ShareFetchData shareFetchData;
+    private final ShareFetchData partitionsToComplete;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsAcquired;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
-            ShareFetchData shareFetchData,
+            ShareFetchData partitionsToComplete,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
-        super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
-        this.shareFetchData = shareFetchData;
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
+        super(partitionsToComplete.fetchParams().maxWaitMs, Option.empty());
+        this.partitionsToComplete = partitionsToComplete;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsAcquired = new LinkedHashMap<>();

Review Comment:
   Why is this a LinkedHashMap instead of just a HashMap?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +147,49 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLog(topicPartitionData);
+                maybeUpdateFetchOffsetMetadata(replicaManagerReadResponse);
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsAcquired = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsAcquired.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsAcquired.clear();

Review Comment:
   We can skip clearing these two maps here since the operation is completed at this point. In `onComplete()`, we don't clear these two maps. So, this will make the behavior more consistent.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +233,139 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
             }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private void maybeUpdateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} errored out for topic partition {}",
+                    replicaManagerLogReadResult, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= partitionsToComplete.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it to support other FetchIsolation types.
+        FetchIsolation isolationType = partitionsToComplete.fetchParams().isolation;
+        if (isolationType == FetchIsolation.LOG_END)
+            return offsetSnapshot.logEndOffset;
+        else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+            return offsetSnapshot.highWatermark;
+        else
+            return offsetSnapshot.lastStableOffset;
+
+    }
+
+    private Map<TopicIdPartition, LogReadResult> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            partitionsToComplete.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UNBOUNDED_QUOTA,
+            true);
+
+        Map<TopicIdPartition, LogReadResult> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            responseData.put(tpLogResult._1(), tpLogResult._2());
+            return BoxedUnit.UNIT;
+        });
+
+        log.trace("Data successfully retrieved by replica manager: {}", responseData);
+        return responseData;
+    }
+
+    private boolean anyTopicIdPartitionHasLogReadError(Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse) {
+        for (LogReadResult logReadResult : replicaManagerReadResponse.values()) {
+            if (logReadResult.error().code()  != Errors.NONE.code()) {
+                return true;
+            }
+        }
+        return false;

Review Comment:
   This can be a bit simpler.
   
   ```
   return replicaManagerReadResponse.values().stream()
       .anyMatch(logReadResult -> logReadResult.error().code() != Errors.NONE.code());
   ```



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +233,139 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
             }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private void maybeUpdateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} errored out for topic partition {}",
+                    replicaManagerLogReadResult, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= partitionsToComplete.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it to support other FetchIsolation types.
+        FetchIsolation isolationType = partitionsToComplete.fetchParams().isolation;
+        if (isolationType == FetchIsolation.LOG_END)
+            return offsetSnapshot.logEndOffset;
+        else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+            return offsetSnapshot.highWatermark;
+        else
+            return offsetSnapshot.lastStableOffset;
+
+    }
+
+    private Map<TopicIdPartition, LogReadResult> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            partitionsToComplete.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UNBOUNDED_QUOTA,
+            true);
+
+        Map<TopicIdPartition, LogReadResult> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            responseData.put(tpLogResult._1(), tpLogResult._2());
+            return BoxedUnit.UNIT;
+        });
+
+        log.trace("Data successfully retrieved by replica manager: {}", responseData);
+        return responseData;
+    }
+
+    private boolean anyTopicIdPartitionHasLogReadError(Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse) {

Review Comment:
   anyTopicIdPartitionHasLogReadError => anyPartitionHasLogReadError ?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +147,49 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLog(topicPartitionData);
+                maybeUpdateFetchOffsetMetadata(replicaManagerReadResponse);
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsAcquired = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsAcquired.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsAcquired.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+                        sharePartitions.keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+                    sharePartitions.keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(topicPartitionData.keySet());

Review Comment:
   This part of the exception is still quite confusing to me. If we hit the exception, should we release the locks before calling forceComplete()? Otherwise, `forceComplete` will try to acquire the same locks again, but can't. It will also help to narrow the try/catch to where the exception can be thrown.



##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -574,6 +575,7 @@ void processShareFetch(ShareFetchData shareFetchData) {
         // Initialize lazily, if required.
         Map<TopicIdPartition, Throwable> erroneous = null;
         Set<DelayedShareFetchKey> delayedShareFetchWatchKeys = new HashSet<>();
+        Map<TopicIdPartition, SharePartition> sharePartitions = new LinkedHashMap<>();

Review Comment:
   Why is this a LinkedHashMap instead of just a HashMap?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +233,139 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();

Review Comment:
   missingFetchOffsetMetadataTopicPartitions => partitionsMissingFetchOffsetMetadata?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -80,64 +89,56 @@ public void onExpiration() {
     @Override
     public void onComplete() {
         log.trace("Completing the delayed share fetch request for group {}, member {}, "
-            + "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
-            topicPartitionDataFromTryComplete.keySet());
+            + "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+            partitionsAcquired.keySet());
 
-        if (shareFetchData.future().isDone())
+        if (partitionsToComplete.future().isDone())
             return;
 
         Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
         // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
+        if (partitionsAcquired.isEmpty())
             topicPartitionData = acquirablePartitions();
         // tryComplete invoked forceComplete, so we can use the data from tryComplete.
         else
-            topicPartitionData = topicPartitionDataFromTryComplete;
+            topicPartitionData = partitionsAcquired;
 
         if (topicPartitionData.isEmpty()) {
             // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
+            partitionsToComplete.future().complete(Collections.emptyMap());
             return;
         }
         log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-            topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
+            topicPartitionData, partitionsToComplete.groupId(), partitionsToComplete.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
-            Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
-            shareFetchData.future().complete(result);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There shouldn't be a case when we have a partitionsAlreadyFetched value here and this variable is getting
+                // updated in a different tryComplete thread.

Review Comment:
   Hmm, Apoorv had a good point in his comment (https://github.com/apache/kafka/pull/17539#discussion_r1828579315). There seems to be a potential problem. It's possible that thread1 calls `tryComplete`, finds `completed` to be false, and is about to set `partitionsAlreadyFetched`. The expiration thread then calls `forceComplete` and sets `completed` to true and proceeds to here. Now, thread1 continues and updates `partitionsAlreadyFetched`. The expiration thread will pick up the wrong `partitionsAlreadyFetched`.



##########
core/src/main/java/kafka/server/share/ShareFetchUtils.java:
##########
@@ -64,11 +64,7 @@ static Map<TopicIdPartition, ShareFetchResponseData.PartitionData> processFetchR
             TopicIdPartition topicIdPartition = entry.getKey();
             FetchPartitionData fetchPartitionData = entry.getValue();
 
-            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
-                continue;
-            }
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);

Review Comment:
   This is an existing issue. In the line below, we are logging for each partition, but shareFetchData contains the full request.
   
   ```
                   log.trace("Acquired records for topicIdPartition: {} with share fetch data: {}, records: {}",
                       topicIdPartition, shareFetchData, shareAcquiredRecords);
   ```



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,130 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testTryCompleteReturnsFalseWhenMinBytesNotSatisfiedOnFirstFetch() {
+        String groupId = "grp";
+        Uuid topicId = Uuid.randomUuid();
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(topicId, new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(topicId, new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        when(sp0.maybeAcquireFetchLock()).thenReturn(true);
+        when(sp1.maybeAcquireFetchLock()).thenReturn(true);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                2, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        when(sp0.canAcquireRecords()).thenReturn(true);
+        when(sp1.canAcquireRecords()).thenReturn(false);
+        when(sp0.acquire(any(), anyInt(), any())).thenReturn(
+            ShareAcquiredRecords.fromAcquiredRecords(new ShareFetchResponseData.AcquiredRecords().setFirstOffset(0).setLastOffset(3).setDeliveryCount((short) 1)));
+
+        // We are testing the case when the share partition is getting fetched for the first time, so for the first time
+        // the fetchOffsetMetadata will return empty. Post the readFromLog call, the fetchOffsetMetadata will be
+        // populated for the share partition, which has 1 as the positional difference, so it doesn't satisfy the minBytes(2).
+        when(sp0.fetchOffsetMetadata())
+            .thenReturn(Optional.empty())
+            .thenReturn(Optional.of(new LogOffsetMetadata(0, 1, 0)));
+        LogOffsetMetadata hwmOffsetMetadata = new LogOffsetMetadata(1, 1, 1);
+        mockTopicIdPartitionFetchBytes(replicaManager, tp0, hwmOffsetMetadata);
+
+        doAnswer(invocation -> buildLogReadResult(Collections.singleton(tp0))).when(replicaManager).readFromLog(any(), any(), any(ReplicaQuota.class), anyBoolean());
+
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
+            .withShareFetchData(shareFetchData)
+            .withSharePartitions(sharePartitions)
+            .withReplicaManager(replicaManager)
+            .build());
+        assertFalse(delayedShareFetch.isCompleted());
+
+        // Since sp1 cannot be acquired, tryComplete should return false.
+        assertFalse(delayedShareFetch.tryComplete());
+        assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(1)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testTryCompleteReturnsFalseWhenMinBytesNotSatisfiedOnLatestFetch() {

Review Comment:
   testTryCompleteReturnsFalseWhenMinBytesNotSatisfiedOnLatestFetch =>
   testTryCompleteWhenMinBytesNotSatisfiedOnSubsequentFetch ?



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,130 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testTryCompleteReturnsFalseWhenMinBytesNotSatisfiedOnFirstFetch() {

Review Comment:
   testTryCompleteReturnsFalseWhenMinBytesNotSatisfiedOnFirstFetch => testTryCompleteWhenMinBytesNotSatisfiedOnFirstFetch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816832978


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();

Review Comment:
   not necessary, it can be null as well. Hence, we use optional here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815586235


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata() == null) {
+                if (replicaManagerReadResponseData == null) {
+                    replicaManagerReadResponseData = readFromLog(topicPartitionData);

Review Comment:
   yeah, but in case there are multiple partitions for which offset metadata is missing, I thought that its better to make a single `readFromLog` call instead of multiple `readFromLog` calls. Thus, using all the topic partitions in one call itself. Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816829826


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -261,6 +263,11 @@ public static RecordState forId(byte id) {
      */
     private long endOffset;
 
+    /**
+     * We maintain the latest fetch offset metadata to estimate the minBytes requirement more efficiently.
+     */
+    private Optional<LogOffsetMetadata> latestFetchOffsetMetadata;

Review Comment:
   I did it because the value can be null, so we thought it would be better to keep it as Optional https://github.com/apache/kafka/pull/17539#discussion_r1815520277



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816657075


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }

Review Comment:
   So this iteration will always be executed for every share fetch when the `missingFetchOffsetMetadataTopicPartitions` will rarely be true, only when a new SharePartition is created. Hence, I was thinking why not to have such update only on SharePartition initialization. Though I understand that current `readFromLog` API requires fethchParams but is there an API which can supply the LogOffsetMetadata when requested with topic partition and specific offset(start offset of share partition)? @junrao wdyt? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1812514360


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1475,6 +1486,10 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    LogOffsetMetadata latestFetchOffsetMetadata() {

Review Comment:
   Is it need to be default scoped?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -303,6 +310,7 @@ public static RecordState forId(byte id) {
         this.partitionState = SharePartitionState.EMPTY;
         this.replicaManager = replicaManager;
         this.groupConfigManager = groupConfigManager;
+        latestFetchOffsetMetadata = null;

Review Comment:
   Isn't by default it will be null, do we need to define it here?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -518,7 +526,8 @@ public long nextFetchOffset() {
      */
     public List<AcquiredRecords> acquire(
         String memberId,
-        FetchPartitionData fetchPartitionData
+        FetchPartitionData fetchPartitionData,
+        LogOffsetMetadata fetchOffsetMetadata

Review Comment:
   You made a new class for this, shouldn't we pass that here?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -532,6 +541,8 @@ public List<AcquiredRecords> acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   So in a fetch result of 10 batches we should store the last batch info, is that correct? If yes then shouldn't name of variable be appropriately defined?



##########
core/src/main/java/kafka/server/share/FetchPartitionOffsetData.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server.share;

Review Comment:
   Can it reside in share module under `fetch`?



##########
core/src/main/java/kafka/server/share/FetchPartitionOffsetData.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server.share;
+
+import org.apache.kafka.server.storage.log.FetchPartitionData;
+import org.apache.kafka.storage.internals.log.LogOffsetMetadata;
+
+import java.util.Objects;
+
+public class FetchPartitionOffsetData {
+    private final FetchPartitionData fetchPartitionData;
+    private final LogOffsetMetadata logOffsetMetadata;
+
+    public FetchPartitionOffsetData(FetchPartitionData fetchPartitionData, LogOffsetMetadata logOffsetMetadata) {
+        this.fetchPartitionData = fetchPartitionData;
+        this.logOffsetMetadata = logOffsetMetadata;
+    }
+
+    public FetchPartitionData fetchPartitionData() {
+        return fetchPartitionData;
+    }
+
+    public LogOffsetMetadata logOffsetMetadata() {
+        return logOffsetMetadata;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        FetchPartitionOffsetData that = (FetchPartitionOffsetData) o;
+        return fetchPartitionData.equals(that.fetchPartitionData) && logOffsetMetadata == that.logOffsetMetadata;
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(fetchPartitionData, logOffsetMetadata);
+    }

Review Comment:
   Why do we need this?



##########
core/src/main/java/kafka/server/share/FetchPartitionOffsetData.java:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server.share;
+
+import org.apache.kafka.server.storage.log.FetchPartitionData;
+import org.apache.kafka.storage.internals.log.LogOffsetMetadata;
+
+import java.util.Objects;
+
+public class FetchPartitionOffsetData {
+    private final FetchPartitionData fetchPartitionData;
+    private final LogOffsetMetadata logOffsetMetadata;
+
+    public FetchPartitionOffsetData(FetchPartitionData fetchPartitionData, LogOffsetMetadata logOffsetMetadata) {
+        this.fetchPartitionData = fetchPartitionData;
+        this.logOffsetMetadata = logOffsetMetadata;
+    }
+
+    public FetchPartitionData fetchPartitionData() {
+        return fetchPartitionData;
+    }
+
+    public LogOffsetMetadata logOffsetMetadata() {
+        return logOffsetMetadata;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        FetchPartitionOffsetData that = (FetchPartitionOffsetData) o;
+        return fetchPartitionData.equals(that.fetchPartitionData) && logOffsetMetadata == that.logOffsetMetadata;
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(fetchPartitionData, logOffsetMetadata);
+    }
+
+    @Override
+    public String toString() {
+        return "FetchPartitionOffsetData(fetchPartitionData=" + fetchPartitionData +

Review Comment:
   Will this log line print anything meaningful? I don't see toString in `FetchPartitionData`. Should we return what's required here?



##########
core/src/main/java/kafka/server/share/ShareFetchUtils.java:
##########
@@ -84,7 +88,8 @@ static Map<TopicIdPartition, ShareFetchResponseData.PartitionData> processFetchR
                     partitionData.setErrorMessage(Errors.NONE.message());
                 }
             } else {
-                List<AcquiredRecords> acquiredRecords = sharePartition.acquire(shareFetchData.memberId(), fetchPartitionData);
+                List<AcquiredRecords> acquiredRecords = sharePartition.acquire(shareFetchData.memberId(), fetchPartitionData,
+                    logOffsetMetadata);

Review Comment:
   Merge with previous line.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +217,29 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   Should it be default scoped? If used for tests then please write // Visible for testing.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +217,29 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        AtomicLong accumulatedSize = new AtomicLong(0);
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   How the exception will be handled with this method? Do we have a test case when exception is thrown by this replica manager call?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1831878254


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +58,25 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsAcquired;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    // The topic partitions that need to be completed for the share fetch request are given by sharePartitions.
+    // sharePartitions is a subset of shareFetchData.
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {

Review Comment:
   Yeah, good point. We should.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1809360360


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +191,62 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        boolean minBytesSatisfied = false;
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);

Review Comment:
   Should it be AtomicLong?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,13 +133,20 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
-        log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
-                "topic partitions {}", shareFetchData.groupId(),
-                shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet());
+        if (!topicPartitionData.isEmpty()) {
+            replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, false);
+            if (!replicaManagerFetchDataFromTryComplete.isEmpty())
+                return forceComplete();
+            log.info("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                shareFetchData.partitionMaxBytes().keySet());
+        } else {
+            log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +

Review Comment:
   Wouldn't this be too common? Should we move it trace?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,13 +133,20 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
-        log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
-                "topic partitions {}", shareFetchData.groupId(),
-                shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet());
+        if (!topicPartitionData.isEmpty()) {
+            replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, false);
+            if (!replicaManagerFetchDataFromTryComplete.isEmpty())
+                return forceComplete();
+            log.info("minBytes is not satisfied for the share fetch request for group {}, member {}, " +

Review Comment:
   info seems to be too much here, how helpful this log would be in info mode? Can we move it to debug please.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);
             shareFetchData.future().complete(result);
         } catch (Exception e) {
             log.error("Error processing delayed share fetch request", e);
             shareFetchData.future().completeExceptionally(e);
         } finally {
             // Releasing the lock to move ahead with the next request in queue.
-            releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            releasePartitionLocks(shareFetchData.groupId(), fetchResponseData.keySet());

Review Comment:
   Will it not depend on `readFromLog` API response i.e. if you sent 3 partitions then is it guaranteed that replica manager will return all 3 partitions in response?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1812914606


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +217,29 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        AtomicLong accumulatedSize = new AtomicLong(0);
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   I've added the handling and test case for this in my latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1818159160


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hi @junrao , agreed with the simpler approach. I have made the following changes in my latest commit- 
   > In the current approach, (1) if any call moves endOffset, we reset the latestFetchOffsetMetadata to Optional.empty()
   
   I reset the `latestFetchOffsetMetadata` to Optional.empty() if -
   1. `acquire()` results in non-empty acquired records in `ShareFetchUtils`.
   2. acquisition lock timeout is called.
   3. release acquired records on session close is called.
   I haven't made the change in `acknowledge()` method of `SharePartition`, since in the common case all the `ACQUIRED` records will moved to `ACKNOWLEDGED` state and endOffset doesn't change then.
   
   > In tryComplete, if latestFetchOffsetMetadata is empty, we call readFromLog and update latestFetchOffsetMetadata
   
   This functionality has been added in previous commits.
   
   Please review my PR whenever you can. Thanks!
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2452271960

   Hi @junrao , i have addressed all your comments. Please re-review my PR when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826561286


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();

Review Comment:
   Hi @junrao , we can do that but then once we are in the `else` of this check -` if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData))`, after we have release the partitions lock, we also need to` partitionsToComplete.clear()`. Hence, I've tried to avoid doing this by  assigning `partitionsToComplete` once we are sure that we can do a `forceComplete()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816619673


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());

Review Comment:
   So the update should be safe with multiple threads as we have acquired the lock on share partition which guards us from 2 threads trying to update the offset metadata. But we should write comments on the share partition update method that the caller of the method should ensure that share partition fetch lock is acquired prior invoking the updateLatestFetchOffsetMetadata.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;

Review Comment:
   Can it be in a separate method i.e. divide methods.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;

Review Comment:
   Why do we satisfy the min byte criteria if share partition is null?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }

Review Comment:
   So this iteration will always be excuted for every share fetch when the `missingFetchOffsetMetadataTopicPartitions` will rarely be true, only when a new SharePartition is created. Hence I was thinking why not to have such update only on SharePartition initilization. Though I understand that current `readFromLog` API requires fethchParams but is there an API which can supply the LogOffsetMetadata when requested with topic partition and specific offset(start offset of share partition)? @junrao wdyt? 



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -517,20 +525,22 @@ public long nextFetchOffset() {
      * @param maxFetchRecords    The maximum number of records that should be acquired, this is a soft
      *                           limit and the method might acquire more records than the maxFetchRecords,
      *                           if the records are already part of the same fetch batch.
-     * @param fetchPartitionData The fetched records for the share partition.
+     * @param fetchPartitionOffsetData The fetched records for the share partition along with log offset metadata

Review Comment:
   Please correct the alignment of comments.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,25 +109,11 @@ public void onComplete() {
                 topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, FetchPartitionOffsetData> responseData;
+            if (logReadResponseFromTryComplete == null || logReadResponseFromTryComplete.isEmpty())

Review Comment:
   Can it be non-null and empty ever i.e. do you require your second condition?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -261,6 +263,11 @@ public static RecordState forId(byte id) {
      */
     private long endOffset;
 
+    /**
+     * We maintain the latest fetch offset metadata to estimate the minBytes requirement more efficiently.
+     */
+    private Optional<LogOffsetMetadata> latestFetchOffsetMetadata;

Review Comment:
   Why it's optional?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            TopicIdPartition topicIdPartition = tpLogResult._1();
+            LogReadResult logResult = tpLogResult._2();
+            FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+            responseData.put(topicIdPartition, new FetchPartitionOffsetData(fetchPartitionData, logResult.info().fetchOffsetMetadata));

Review Comment:
   So if we are fetch from offset 0 and gets response from log for 0-1000 offsets then `logResult.info().fetchOffsetMetadata` contains information for 0 offset or 1000th offset i.e. which offset metadata does it hold?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();

Review Comment:
   Should the varibale in SharePartition be Optional? We always should have that, correct?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,13 +145,28 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
-        log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
-                "topic partitions {}", shareFetchData.groupId(),
-                shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet());
+        if (!topicPartitionData.isEmpty()) {
+            MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataResult =
+                maybeUpdateFetchOffsetMetadataForTopicPartitions(topicPartitionData);

Review Comment:
   Merge the lines.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            TopicIdPartition topicIdPartition = tpLogResult._1();
+            LogReadResult logResult = tpLogResult._2();
+            FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+            responseData.put(topicIdPartition, new FetchPartitionOffsetData(fetchPartitionData, logResult.info().fetchOffsetMetadata));
+            return BoxedUnit.UNIT;
+        });
+
+        log.trace("Data successfully retrieved by replica manager: {}", responseData);
+        return responseData;
+    }
+
+    // Visible for testing.
+    Map<TopicIdPartition, FetchPartitionOffsetData> combineLogReadResponseForOnComplete(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingLogReadTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            if (!logReadResponseFromTryComplete.containsKey(topicIdPartition)) {
+                missingLogReadTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });

Review Comment:
   Again this will be rare hence shall we delay initiliazing LinkedHashMap.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;

Review Comment:
   Sorry, I didn't understand when we can have the offset in share partition > partition end offset?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;

Review Comment:
   Can it go in a method please.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,25 +109,11 @@ public void onComplete() {
                 topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, FetchPartitionOffsetData> responseData;
+            if (logReadResponseFromTryComplete == null || logReadResponseFromTryComplete.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                responseData = combineLogReadResponseForOnComplete(topicPartitionData);

Review Comment:
   Can it ever occur that you have non-empty `logReadResponseFromTryComplete` but `topicPartitionData` came from fresh aquisition from line 98 (topicPartitionData = acquirablePartitions();). I think never, can you just write comments for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828852012


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +

Review Comment:
   yes, the log statement is correct because because `isMinBytesSatisfied` can run only if `anyTopicIdPartitionHasLogReadError` returns false. We should only check `isMinBytesSatisfied` if `anyTopicIdPartitionHasLogReadError` returns false. If `anyTopicIdPartitionHasLogReadError` return true, then we do a `forceComplete`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828860825


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);

Review Comment:
   you're right, I've changed the `releasePartitionLocks(topicPartitionData.keySet())`. If the `forceComplete` is successful, then the partition locks are released from `onComplete` finally block and it doesn't concern here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828877645


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,34 +113,27 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There can't be a case when we have a non-empty logReadResponse, and we have a fresh topicPartitionData
+                // using tryComplete because purgatory ensures that 2 tryCompletes calls do not happen at the same time.

Review Comment:
   What I mean is that `partitionsAlreadyFetched` value can't be changed once we enter this point in `onComplete` via `forceComplete` either by expiration or by a `tryComplete` successful call. IIUC, `forceComplete` uses this `AtomicBoolean` variable `completed` which is used as locking mechanism for `forceComplete`. This should ensure atomicity of global variables between `tryComplete` and `forceComplete` we use in `DelayedShareFetch`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830454251


##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -574,6 +575,7 @@ void processShareFetch(ShareFetchData shareFetchData) {
         // Initialize lazily, if required.
         Map<TopicIdPartition, Throwable> erroneous = null;
         Set<DelayedShareFetchKey> delayedShareFetchWatchKeys = new HashSet<>();
+        Map<TopicIdPartition, SharePartition> sharePartitions = new LinkedHashMap<>();

Review Comment:
   Hi Jun, reading online regarding the performance of LinkedHashMap and HashMap - LinkedHashmap offers better performance when iterating through elements since they maintain an ordered entry list, while a HashMap offers better performance when accessing large datasets. Furthermore, when storing objects, LinkedHashmap stores objects in key-value pairs, while HashMap stores them in hash table. The type of key used also affects performance. 
   
   Since now we are iterating over `sharePartitions` within the function `acquirablePartitions`, I thought it would be more efficient to use LinkedHashMap over HashMap.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1831742440


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,42 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
-
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
-            boolean completedByMe = forceComplete();
-            // If invocation of forceComplete is not successful, then that means the request is already completed
-            // hence release the acquired locks.
-            if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLog(topicPartitionData);
+                maybeUpdateFetchOffsetMetadata(replicaManagerReadResponse);
+                if (anyPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsAcquired = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsAcquired.keySet());
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        sharePartitions.keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    sharePartitions.keySet());
             }
-            return completedByMe;
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+            releasePartitionLocks(topicPartitionData.keySet());

Review Comment:
   Should we reset partitionsAcquired and partitionsAlreadyFetched?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +228,135 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> partitionsMissingFetchOffsetMetadata = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                partitionsMissingFetchOffsetMetadata.put(topicIdPartition, partitionData);
             }
+        });
+        if (partitionsMissingFetchOffsetMetadata.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(partitionsMissingFetchOffsetMetadata);
+    }
+
+    private void maybeUpdateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} errored out for topic partition {}",
+                    replicaManagerLogReadResult, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   Let's add a comment that the minBytes estimation currently assumes the common case where all fetched data are acquirable. 



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +58,25 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsAcquired;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    // The topic partitions that need to be completed for the share fetch request are given by sharePartitions.
+    // sharePartitions is a subset of shareFetchData.
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {

Review Comment:
   If ordering is important, should we explicitly define it as LinkedHashMap in all the places?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830454251


##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -574,6 +575,7 @@ void processShareFetch(ShareFetchData shareFetchData) {
         // Initialize lazily, if required.
         Map<TopicIdPartition, Throwable> erroneous = null;
         Set<DelayedShareFetchKey> delayedShareFetchWatchKeys = new HashSet<>();
+        Map<TopicIdPartition, SharePartition> sharePartitions = new LinkedHashMap<>();

Review Comment:
   Hi Jun, reading online regarding the performance of LinkedHashMap and HashMap - Linkedhashmap offers better performance when iterating through elements since they maintain an ordered entry list, while a Hashmap offers better performance when accessing large datasets. Furthermore, when storing objects, Linkedhashmap stores objects in key-value pairs, while Hashmap stores them in hash table. The type of key used also affects performance. 
   
   Since now we are iterating over `sharePartitions` within the function `acquirablePartitions`, I thought it would be more efficient to use LinkedHashMap over HashMap.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1824937560


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -65,6 +70,7 @@ public class DelayedShareFetch extends DelayedOperation {
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
         this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.logReadResponse = new LinkedHashMap<>();

Review Comment:
   topicPartitionDataFromTryComplete => partitionsToComplete ?
   logReadResponse => partitionsAlreadyFetched ?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,16 +145,34 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
-
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
-            boolean completedByMe = forceComplete();
-            // If invocation of forceComplete is not successful, then that means the request is already completed
-            // hence release the acquired locks.
-            if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   We are still calling `sharePartitionManager.sharePartition` in multiple places (`anySharePartitionNoLongerExists`, `acquirablePartitions`, `isMinBytesSatisfied` and `maybeUpdateFetchOffsetMetadataForTopicPartitions`), each of which needs to handle null SharePartition since the sharePartition could disappear any time. I was thinking that we could get all sharePartitions once at the beginning and pass them around to other methods. This way, the null handling is only done once.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,7 +224,158 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return null;
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(replicaManagerReadResponseData);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadataForMissingTopicPartitions(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it other FetchIsolation types.
+        FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+        if (isolationType == FetchIsolation.LOG_END)
+            return offsetSnapshot.logEndOffset;
+        else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+            return offsetSnapshot.highWatermark;
+        else
+            return offsetSnapshot.lastStableOffset;
+
+    }
+
+    private Map<TopicIdPartition, LogReadResult> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UNBOUNDED_QUOTA,
+            true);
+
+        Map<TopicIdPartition, LogReadResult> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            responseData.put(tpLogResult._1(), tpLogResult._2());
+            return BoxedUnit.UNIT;
+        });
+
+        log.trace("Data successfully retrieved by replica manager: {}", responseData);
+        return responseData;
+    }
+
+    private boolean anySharePartitionNoLongerExists() {
+        for (TopicIdPartition topicIdPartition: shareFetchData.partitionMaxBytes().keySet()) {
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                return true;
+            }
+        }
+        return false;
+    }
+
+    // Visible for testing.
+    Map<TopicIdPartition, LogReadResult> combineLogReadResponse(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingLogReadTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            if (!logReadResponse.containsKey(topicIdPartition)) {
+                missingLogReadTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        }

Review Comment:
   This code can be a bit more concise.
   
   ```
   topicPartitionData.forEach((topicIdPartition, partitionData) -> {
       if (!logReadResponse.containsKey(topicIdPartition)) {
           missingLogReadTopicPartitions.put(topicIdPartition, partitionData);
       }
   });
   ```



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,7 +224,158 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   maybeUpdateFetchOffsetMetadataForTopicPartitions => maybeReadFromLogAndUpdateFetchOffsetMetadata ?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,7 +224,158 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private Map<TopicIdPartition, LogReadResult> maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return null;

Review Comment:
   Could we just return an empty map? This way, the caller doesn't need to do the null check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828831006


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {

Review Comment:
   I have been asked to add else blocks in the above comments, hence I don' think I should change it again. https://github.com/apache/kafka/pull/17539#discussion_r1819610603



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2455446039

   Hi @junrao , I have addressed your comments. Please re-review my PR when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828820632


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Hi @junrao, its not always a guarantee that `partitionsToComplete` has been set when we hit this exception, but even in that case this line of code shouldn't be a problem since we are are initializing it as an empty map and also we clear the map if the invocation of `forceComplete` is unsuccessful post `isMinBytesSatisfied` check. So, I can add a check that partitionsToComplete is not empty, then call this line, but I don't think even this line is a problem. Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828845131


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {

Review Comment:
   Since, I am using continue in the `loop`, I prefer to do it using for instead of forEach.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {

Review Comment:
   Since, I am using `continue` in the loop, I prefer to do it using for instead of forEach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828877645


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,34 +113,27 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There can't be a case when we have a non-empty logReadResponse, and we have a fresh topicPartitionData
+                // using tryComplete because purgatory ensures that 2 tryCompletes calls do not happen at the same time.

Review Comment:
   What I mean is that `partitionsAlreadyFetched` value can't be changed once we enter this point in `onComplete` via `forceComplete` either by expiration or by a `tryComplete` successful call. IIUC, `forceComplete` uses this `AtomicBoolean` variable `completed` which is used as locking mechanism for `forceComplete`. This should ensure atomicity of global variables between `tryComplete` and `forceComplete` we use in `DelayedShareFetch`. I'll change the comment to explain this better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828902626


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   yes, I'll remove `anySharePartitionNoLongerExists()` and also iterate on `sharePartitions` rather than using `shareFetchData.partitionMaxBytes()`. Will remove the null checks from our code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1829256794


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+        return replicaManagerReadResponseData;
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   logged a JIRA https://issues.apache.org/jira/browse/KAFKA-17943 for the same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806725239


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }
+        }
+        // This call is coming from onComplete, hence we return the response data irrespective of whether minBytes is
+        // satisfied or not.
+        if (hasRequestTimedOut)
+            return responseData;

Review Comment:
   Now that I think again,  I should return an empty map since we have not been able to satisfy all the fetch request criterias. @junrao your thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806729605


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);

Review Comment:
   yeah, since `processFetchResponse` can handle it, that's why I didn't add any check here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816512680


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   mkaes sense, IIUC, whenever there is an acknowledge or acquisition lock timeout or release records on session close, I'll `latestFetchOffsetMetadata` in `SharePartition` to `Optional.empty()` in my latest commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2433136716

   Hi @junrao @apoorvmittal10 , thanks for being kind enough to reviewing this PR. I have addressed both your comments. Please take a look when you can!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815739504


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hmm, to me `acquire` seems to be the wrong place to update `latestFetchOffsetMetadata`. We update `latestFetchOffsetMetadata` with the fetch offset and then acquire a few batches after the fetch offset. This typically moves the next fetch offset, which makes `latestFetchOffsetMetadata` useless for the next fetch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806720826


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,10 +133,13 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
+        if (!topicPartitionData.isEmpty()) {
+            replicaManagerFetchDataFromTryComplete = replicaManagerFetchData(topicPartitionData, false);
+            if (!replicaManagerFetchDataFromTryComplete.isEmpty())
+                return forceComplete();
+        }
         log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +

Review Comment:
   You're right... I'll change it to `
   log.info("Fetch cannot be completed for the partitions in the share fetch request for group {}, member {}, " +
                   "topic partitions {}", shareFetchData.groupId(),
                   shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet())
   `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806722986


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();

Review Comment:
   yeah, it makes sense. I'll make the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1812784514


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -532,6 +541,8 @@ public List<AcquiredRecords> acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   we store the last fetched offset's properties in `LogOffsetMetadata` class like the message offset, file position etc. Hence, naming it `latestFetchOffsetMetadata` makes more sense than `latestFetchBatchMetadata`. Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1807312461


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;

Review Comment:
   I can get rid of it, but then the variable name `replicaManagerFetchDataFromTryComplete` won't make sense if its getting some values in `onComplete`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1809417014


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -64,8 +64,8 @@ public class DelayedShareFetch extends DelayedOperation {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
         this.sharePartitionManager = sharePartitionManager;
+        this.replicaManagerFetchDataFromTryComplete = new HashMap<>();

Review Comment:
   Should we maintain the order by `LinkedHashMap` as earlier?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815599413


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   IIUC, everytime we fetch new records from `readFromLog`, I utilize the `LogOffsetMetadata` object from the response to update the `latestFetchOffsetMetadata` object while doing the acquire. Hence, this way I have the most recent fetched offset's metadata that I always update through `acquire` method. Most of the time, it is going to be the endOffset's metadata itself, as soon as new data is produced to the topic partition. Anything else that I need to do here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1819587266


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1538,27 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {

Review Comment:
   Could we make fetchOffsetMetadata Optional instead of relying on `null`?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions == null || missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new FetchOffsetMetadataUpdateResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(missingFetchOffsetMetadataTopicPartitions, replicaManagerReadResponseData);
+    }
+
+    private FetchOffsetMetadataUpdateResult updateFetchOffsetMetadataForMissingTopicPartitions(
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions,
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.getOrDefault(topicIdPartition, null);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new FetchOffsetMetadataUpdateResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it other FetchIsolation types.
+        FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+        if (isolationType == FetchIsolation.LOG_END)
+            return offsetSnapshot.logEndOffset;
+        else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+            return offsetSnapshot.highWatermark;
+        else
+            return offsetSnapshot.lastStableOffset;
+
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();

Review Comment:
   Do we need this wrapper class `FetchPartitionOffsetData`? It seems that it's simpler for `readFromLog` to `return Map<TopicIdPartition, LogReadResult>`. We can then convert `LogReadResult` to `Map<TopicIdPartition, ShareFetchResponseData.PartitionData>` in `onComplete`.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -149,16 +148,32 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
-
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
-            boolean completedByMe = forceComplete();
-            // If invocation of forceComplete is not successful, then that means the request is already completed
-            // hence release the acquired locks.
-            if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+
+        if (!topicPartitionData.isEmpty()) {
+            FetchOffsetMetadataUpdateResult fetchOffsetMetadataUpdateResult = maybeUpdateFetchOffsetMetadataForTopicPartitions(topicPartitionData);
+            if (isMinBytesSatisfied(topicPartitionData)) {
+                topicPartitionDataFromTryComplete = topicPartitionData;
+                if (fetchOffsetMetadataUpdateResult.isFetchOffsetMetadataUpdated
+                    && fetchOffsetMetadataUpdateResult.replicaManagerReadResponse != null
+                    && !fetchOffsetMetadataUpdateResult.replicaManagerReadResponse.isEmpty())
+                    logReadResponse = fetchOffsetMetadataUpdateResult.replicaManagerReadResponse;
+                boolean completedByMe = forceComplete();
+                // If invocation of forceComplete is not successful, then that means the request is already completed
+                // hence release the acquired locks.
+                if (!completedByMe) {
+                    releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                }
+                return completedByMe;
             }
-            return completedByMe;
+            log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +

Review Comment:
   It's probably clearer to put those in an `else` clause?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -149,16 +148,32 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
-
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
-            boolean completedByMe = forceComplete();
-            // If invocation of forceComplete is not successful, then that means the request is already completed
-            // hence release the acquired locks.
-            if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+
+        if (!topicPartitionData.isEmpty()) {
+            FetchOffsetMetadataUpdateResult fetchOffsetMetadataUpdateResult = maybeUpdateFetchOffsetMetadataForTopicPartitions(topicPartitionData);
+            if (isMinBytesSatisfied(topicPartitionData)) {
+                topicPartitionDataFromTryComplete = topicPartitionData;
+                if (fetchOffsetMetadataUpdateResult.isFetchOffsetMetadataUpdated

Review Comment:
   This condition seems unnecessary. We need to set `logReadResponse` as long as `fetchOffsetMetadataUpdateResult.replicaManagerReadResponse` is not empty, right?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions == null || missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new FetchOffsetMetadataUpdateResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(missingFetchOffsetMetadataTopicPartitions, replicaManagerReadResponseData);

Review Comment:
   Every passed in partition to `readFromLog` will be included in the response. So, there is no need to pass in both `missingFetchOffsetMetadataTopicPartitions` and `replicaManagerReadResponseData`. We do want to check the error code for each partition. In regular fetch, if any partition has an error code, we send a response immediately. We can just do the same here. 



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions == null || missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new FetchOffsetMetadataUpdateResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(missingFetchOffsetMetadataTopicPartitions, replicaManagerReadResponseData);
+    }
+
+    private FetchOffsetMetadataUpdateResult updateFetchOffsetMetadataForMissingTopicPartitions(
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions,
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {

Review Comment:
   Could we  just make a single `sharePartitionManager.sharePartition` call per `tryComplete` to avoid having to check null repeatedly?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();

Review Comment:
   We could just initialize `missingFetchOffsetMetadataTopicPartitions` with `new LinkedHashMap<>()`. Ditto in `combineLogReadResponse`.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,27 +109,20 @@ public void onComplete() {
             topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
+            Map<TopicIdPartition, FetchPartitionOffsetData> responseData;
+            if (logReadResponse == null)

Review Comment:
   Hmm, it would be better for `logReadResponse` to only be empty, but never `null`.



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -261,6 +263,11 @@ public static RecordState forId(byte id) {
      */
     private long endOffset;
 
+    /**
+     * We maintain the latest fetch offset metadata to estimate the minBytes requirement more efficiently.
+     */
+    private Optional<LogOffsetMetadata> latestFetchOffsetMetadata;

Review Comment:
   latestFetchOffsetMetadata => fetchOffsetMetadata?



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -783,6 +790,12 @@ && checkForStartOffsetWithinBatch(inFlightBatch.firstOffset(), inFlightBatch.las
         } finally {
             lock.writeLock().unlock();
         }
+
+        if (!stateBatches.isEmpty()) {
+            // The next fetch offset will change on release of acquired records on session close, hence we update latestFetchOffsetMetadata
+            // for the share partition.
+            updateLatestFetchOffsetMetadata(null);

Review Comment:
   This is getting a bit hard to track since we need to make this call in all places where endOffset changes. Could we have a method for updating both `endOffset` and `latestFetchOffsetMetadata`? Then we can replace all code that changes `endOffset` with this method.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;

Review Comment:
   I thought we agreed that we want to send a response immediately if a sharePartition can't be found. Is that handled?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -210,7 +225,169 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private FetchOffsetMetadataUpdateResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty()) {
+                if (missingFetchOffsetMetadataTopicPartitions == null)
+                    missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+            }
+        }
+        if (missingFetchOffsetMetadataTopicPartitions == null || missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new FetchOffsetMetadataUpdateResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+        return updateFetchOffsetMetadataForMissingTopicPartitions(missingFetchOffsetMetadataTopicPartitions, replicaManagerReadResponseData);
+    }
+
+    private FetchOffsetMetadataUpdateResult updateFetchOffsetMetadataForMissingTopicPartitions(
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions,
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.getOrDefault(topicIdPartition, null);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new FetchOffsetMetadataUpdateResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());

Review Comment:
   Ideally, we need to handle the exception at the partition level in the caller.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1832915380


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +230,136 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> partitionsMissingFetchOffsetMetadata = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                partitionsMissingFetchOffsetMetadata.put(topicIdPartition, partitionData);
             }
+        });
+        if (partitionsMissingFetchOffsetMetadata.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(partitionsMissingFetchOffsetMetadata);
+    }
+
+    private void maybeUpdateFetchOffsetMetadata(
+        Map<TopicIdPartition, LogReadResult> replicaManagerReadResponseData) {
+        for (Map.Entry<TopicIdPartition, LogReadResult> entry : replicaManagerReadResponseData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            LogReadResult replicaManagerLogReadResult = entry.getValue();
+            if (replicaManagerLogReadResult.error().code() != Errors.NONE.code()) {
+                log.debug("Replica manager read log result {} errored out for topic partition {}",
+                    replicaManagerLogReadResult, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateFetchOffsetMetadata(Optional.of(replicaManagerLogReadResult.info().fetchOffsetMetadata));
+        }
+    }
+
+    // minByes estimation currently assumes the common case where all fetched data is acquirable.
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            FetchRequest.PartitionData partitionData = entry.getValue();
+            LogOffsetMetadata endOffsetMetadata = endOffsetMetadataForTopicPartition(topicIdPartition);
+
+            if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+
+            Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.fetchOffsetMetadata();
+            if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                continue;
+            LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+            if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                    "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                return true;
+            } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                    // This can happen when the fetch operation is falling behind the current segment or the partition
+                    // has just rolled a new segment.
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                        "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                    // we take the partition fetch size as upper bound when accumulating the bytes.
+                    long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                    accumulatedSize += bytesAvailable;
+                }
+            }
+        }
+        return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+    }
+
+    private LogOffsetMetadata endOffsetMetadataForTopicPartition(TopicIdPartition topicIdPartition) {
+        Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+        LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+        // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+        // extend it to support other FetchIsolation types.
+        FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+        if (isolationType == FetchIsolation.LOG_END)
+            return offsetSnapshot.logEndOffset;
+        else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+            return offsetSnapshot.highWatermark;
+        else
+            return offsetSnapshot.lastStableOffset;
+
+    }
+
+    private Map<TopicIdPartition, LogReadResult> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   Since the ordering is important, let's use LinkedHashMap.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +230,136 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
-            if (sharePartition == null) {
-                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), tp);
-                return;
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   Since the ordering is important, let's use LinkedHashMap.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,44 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
-
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
-            boolean completedByMe = forceComplete();
-            // If invocation of forceComplete is not successful, then that means the request is already completed
-            // hence release the acquired locks.
-            if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();

Review Comment:
   Since the ordering is important, let's use LinkedHashMap.



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1576,6 +1582,32 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    // The caller of this function is expected to hold lock.writeLock() when calling this method.
+    protected void updateEndOffsetAndResetFetchOffsetMetadata(long updatedEndOffset) {
+        endOffset = updatedEndOffset;
+        fetchOffsetMetadata = Optional.empty();
+    }
+
+    protected void updateFetchOffsetMetadata(Optional<LogOffsetMetadata> fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            this.fetchOffsetMetadata = fetchOffsetMetadata;
+        } finally {
+            lock.writeLock().unlock();
+        }
+    }
+
+    protected Optional<LogOffsetMetadata> fetchOffsetMetadata() {
+        lock.readLock().lock();
+        try {
+            if (findNextFetchOffset.get())
+                return Optional.empty();

Review Comment:
   This means `tryComplete` will never get non-empty `fetchOffsetMetadata` and its calculation of minBytes will be off. We need to think through how to address this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830410313


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +147,49 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = maybeReadFromLog(topicPartitionData);
+                maybeUpdateFetchOffsetMetadata(replicaManagerReadResponse);
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsAcquired = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsAcquired.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsAcquired.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+                        sharePartitions.keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+                    sharePartitions.keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(topicPartitionData.keySet());

Review Comment:
   makes sense, I've changed the catch block to 
   ```
   catch (Exception e) {
               log.error("Error processing delayed share fetch request", e);
               releasePartitionLocks(topicPartitionData.keySet());
               return forceComplete();
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828254033


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1576,6 +1582,30 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    // The caller of this function is expected to hold lock.writeLock() when calling this method.
+    protected void updateEndOffsetAndResetFetchOffsetMetadata(long updatedEndOffset) {
+        endOffset = updatedEndOffset;
+        fetchOffsetMetadata = Optional.empty();
+    }
+
+    protected void updateFetchOffsetMetadata(Optional<LogOffsetMetadata> fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            this.fetchOffsetMetadata = fetchOffsetMetadata;
+        } finally {
+            lock.writeLock().unlock();
+        }
+    }
+
+    protected Optional<LogOffsetMetadata> fetchOffsetMetadata() {
+        lock.readLock().lock();
+        try {
+            return fetchOffsetMetadata;

Review Comment:
   We need to return an optional `fetchOffsetMetadata` if the value returned from `nextFetchOffset` changes. If `findNextFetchOffset` is false, `nextFetchOffset` returns a value based on endOffset. This case is already covered in this PR. If `findNextFetchOffset` is true, `nextFetchOffset` returns a value not depending on endOffset. So, we should return empty here if `findNextFetchOffset` is true.



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,125 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testDelayedShareFetchTryCompleteReturnsFalseDueToMinBytesOnFirstSharePartitionFetch() {
+        String groupId = "grp";
+        Uuid topicId = Uuid.randomUuid();
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(topicId, new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(topicId, new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        when(sp0.maybeAcquireFetchLock()).thenReturn(true);
+        when(sp1.maybeAcquireFetchLock()).thenReturn(true);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                2, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        when(sp0.canAcquireRecords()).thenReturn(true);
+        when(sp1.canAcquireRecords()).thenReturn(false);
+        when(sp0.acquire(any(), anyInt(), any())).thenReturn(
+            ShareAcquiredRecords.fromAcquiredRecords(new ShareFetchResponseData.AcquiredRecords().setFirstOffset(0).setLastOffset(3).setDeliveryCount((short) 1)));
+
+        // We are testing the case when the share partition is getting fetched for the first time, hence we are using 1
+        // as the file position, so it doesn't satisfy the minBytes(2).
+        when(sp0.fetchOffsetMetadata()).thenReturn(Optional.of(new LogOffsetMetadata(0, 1, 0)));
+        LogOffsetMetadata hwmOffsetMetadata = new LogOffsetMetadata(1, 1, 1);
+        mockTopicIdPartitionFetchBytes(replicaManager, tp0, hwmOffsetMetadata);
+
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
+            .withShareFetchData(shareFetchData)
+            .withSharePartitions(sharePartitions)
+            .withReplicaManager(replicaManager)
+            .build());
+        assertFalse(delayedShareFetch.isCompleted());
+
+        // Since sp1 can be acquired, tryComplete should return true.

Review Comment:
   This comment doesn't match the code.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   It seems that sharePartitions is always a subset of shareFetchData.partitionMaxBytes()? If that's the case, I agree that we don't need anySharePartitionNoLongerExists. However, it would be useful to make sure that the caller passes in sharePartitions and shareFetchData.partitionMaxBytes() with the same set of partition keys.



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -385,6 +492,108 @@ public void testForceCompleteTriggersDelayedActionsQueue() {
         assertFalse(delayedShareFetch1.isCompleted());
         Mockito.verify(replicaManager, times(1)).addToActionQueue(any());
         Mockito.verify(replicaManager, times(0)).tryCompleteActions();
+        Mockito.verify(delayedShareFetch2, times(1)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testCombineLogReadResponse() {
+        String groupId = "grp";
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(Uuid.randomUuid(), new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        CompletableFuture<Map<TopicIdPartition, ShareFetchResponseData.PartitionData>> future = new CompletableFuture<>();
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                1, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            future, partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+            .withShareFetchData(shareFetchData)
+            .withReplicaManager(replicaManager)
+            .withSharePartitions(sharePartitions)
+            .build();
+
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = new HashMap<>();
+        topicPartitionData.put(tp0, mock(FetchRequest.PartitionData.class));
+        topicPartitionData.put(tp1, mock(FetchRequest.PartitionData.class));
+
+        // Case 1 - logReadResponse contains tp0.
+
+        Map<TopicIdPartition, LogReadResult> logReadResponse = Collections.singletonMap(
+            tp0, mock(LogReadResult.class));
+        delayedShareFetch.updateLogReadResponse(logReadResponse);
+
+        doAnswer(invocation -> buildLogReadResult(Collections.singleton(tp1))).when(replicaManager).readFromLog(any(), any(), any(ReplicaQuota.class), anyBoolean());
+        Map<TopicIdPartition, LogReadResult> combinedLogReadResponse = delayedShareFetch.combineLogReadResponse(topicPartitionData);

Review Comment:
   Could we change `combineLogReadResponse` to also take `partitionsAlreadyFetched`? This way, we can get rid of `delayedShareFetch.updateLogReadResponse`.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Hmm, when we hit an exception, do we guarantee that `partitionsToComplete` has been set?



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,125 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testDelayedShareFetchTryCompleteReturnsFalseDueToMinBytesOnFirstSharePartitionFetch() {
+        String groupId = "grp";
+        Uuid topicId = Uuid.randomUuid();
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(topicId, new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(topicId, new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        when(sp0.maybeAcquireFetchLock()).thenReturn(true);
+        when(sp1.maybeAcquireFetchLock()).thenReturn(true);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                2, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        when(sp0.canAcquireRecords()).thenReturn(true);
+        when(sp1.canAcquireRecords()).thenReturn(false);
+        when(sp0.acquire(any(), anyInt(), any())).thenReturn(
+            ShareAcquiredRecords.fromAcquiredRecords(new ShareFetchResponseData.AcquiredRecords().setFirstOffset(0).setLastOffset(3).setDeliveryCount((short) 1)));
+
+        // We are testing the case when the share partition is getting fetched for the first time, hence we are using 1
+        // as the file position, so it doesn't satisfy the minBytes(2).
+        when(sp0.fetchOffsetMetadata()).thenReturn(Optional.of(new LogOffsetMetadata(0, 1, 0)));

Review Comment:
   Hmm, not sure how this test is different from the next one. If this is testing fetching for the first time, sp0.fetchOffsetMetadata() should return empty, right?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return Collections.emptyMap();
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        return readFromLog(missingFetchOffsetMetadataTopicPartitions);
+    }
+
+    private Map<TopicIdPartition, LogReadResult> updateFetchOffsetMetadata(

Review Comment:
   It's kind of weird for this method to return the input. It's more natural for this method to return nothing.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;

Review Comment:
   shareFetchData => partitionsToComplete ?
   partitionsToComplete => partitionsAcquired ?



##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,125 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testDelayedShareFetchTryCompleteReturnsFalseDueToMinBytesOnFirstSharePartitionFetch() {

Review Comment:
   Quite a long name. How about sth like testTryCompleteReturnsFalseWhenMinBytesNotSatisfied?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828775451


##########
core/src/test/java/kafka/server/share/DelayedShareFetchTest.java:
##########
@@ -107,23 +109,125 @@ public void testDelayedShareFetchTryCompleteReturnsFalse() {
         when(sp0.maybeAcquireFetchLock()).thenReturn(true);
         when(sp1.maybeAcquireFetchLock()).thenReturn(true);
 
-        SharePartitionManager sharePartitionManager = mock(SharePartitionManager.class);
-        when(sharePartitionManager.sharePartition(groupId, tp0)).thenReturn(sp0);
-        when(sharePartitionManager.sharePartition(groupId, tp1)).thenReturn(sp1);
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
 
         ShareFetchData shareFetchData = new ShareFetchData(FETCH_PARAMS, groupId, Uuid.randomUuid().toString(),
             new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
 
         when(sp0.canAcquireRecords()).thenReturn(false);
         when(sp1.canAcquireRecords()).thenReturn(false);
-        DelayedShareFetch delayedShareFetch = DelayedShareFetchBuilder.builder()
+        DelayedShareFetch delayedShareFetch = spy(DelayedShareFetchBuilder.builder()
             .withShareFetchData(shareFetchData)
-            .withSharePartitionManager(sharePartitionManager)
-            .build();
+            .withSharePartitions(sharePartitions)
+            .build());
 
         // Since there is no partition that can be acquired, tryComplete should return false.
         assertFalse(delayedShareFetch.tryComplete());
         assertFalse(delayedShareFetch.isCompleted());
+        Mockito.verify(delayedShareFetch, times(0)).releasePartitionLocks(any());
+    }
+
+    @Test
+    public void testDelayedShareFetchTryCompleteReturnsFalseDueToMinBytesOnFirstSharePartitionFetch() {
+        String groupId = "grp";
+        Uuid topicId = Uuid.randomUuid();
+        ReplicaManager replicaManager = mock(ReplicaManager.class);
+        TopicIdPartition tp0 = new TopicIdPartition(topicId, new TopicPartition("foo", 0));
+        TopicIdPartition tp1 = new TopicIdPartition(topicId, new TopicPartition("foo", 1));
+        Map<TopicIdPartition, Integer> partitionMaxBytes = new HashMap<>();
+        partitionMaxBytes.put(tp0, PARTITION_MAX_BYTES);
+        partitionMaxBytes.put(tp1, PARTITION_MAX_BYTES);
+
+        SharePartition sp0 = mock(SharePartition.class);
+        SharePartition sp1 = mock(SharePartition.class);
+
+        when(sp0.maybeAcquireFetchLock()).thenReturn(true);
+        when(sp1.maybeAcquireFetchLock()).thenReturn(true);
+
+        Map<TopicIdPartition, SharePartition> sharePartitions = new HashMap<>();
+        sharePartitions.put(tp0, sp0);
+        sharePartitions.put(tp1, sp1);
+
+        ShareFetchData shareFetchData = new ShareFetchData(
+            new FetchParams(ApiKeys.SHARE_FETCH.latestVersion(), FetchRequest.ORDINARY_CONSUMER_ID, -1, MAX_WAIT_MS,
+                2, 1024 * 1024, FetchIsolation.HIGH_WATERMARK, Optional.empty()), groupId, Uuid.randomUuid().toString(),
+            new CompletableFuture<>(), partitionMaxBytes, MAX_FETCH_RECORDS);
+
+        when(sp0.canAcquireRecords()).thenReturn(true);
+        when(sp1.canAcquireRecords()).thenReturn(false);
+        when(sp0.acquire(any(), anyInt(), any())).thenReturn(
+            ShareAcquiredRecords.fromAcquiredRecords(new ShareFetchResponseData.AcquiredRecords().setFirstOffset(0).setLastOffset(3).setDeliveryCount((short) 1)));
+
+        // We are testing the case when the share partition is getting fetched for the first time, hence we are using 1
+        // as the file position, so it doesn't satisfy the minBytes(2).
+        when(sp0.fetchOffsetMetadata()).thenReturn(Optional.of(new LogOffsetMetadata(0, 1, 0)));

Review Comment:
   my bad, you're right, I've adjusted the mock to return `Optional.empty()` for the first time and `Optional.of(new LogOffsetMetadata(0, 1, 0))` for the second time (post `readFromLog` call)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1831062681


##########
core/src/main/java/kafka/server/share/SharePartitionManager.java:
##########
@@ -574,6 +575,7 @@ void processShareFetch(ShareFetchData shareFetchData) {
         // Initialize lazily, if required.
         Map<TopicIdPartition, Throwable> erroneous = null;
         Set<DelayedShareFetchKey> delayedShareFetchWatchKeys = new HashSet<>();
+        Map<TopicIdPartition, SharePartition> sharePartitions = new LinkedHashMap<>();

Review Comment:
   The reason I suggested to use LinkedHashMap was to maintain the fetch order of partitions.
   
   As per KIP - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070434#KIP932:QueuesforKafka-Fetchingandacknowledgingrecords
   
   > In order to ensure no share-partitions are starved from records being fetched, the share-partition leader rotates the order of share-partitions for which it returns partition information. This ensures that it eventually returns data about all partitions for which data is available.
   
   I will add the rotation in SharePartitionManager to ensure the behaviour.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1824747173


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;

Review Comment:
   Hi @junrao , I had updated it at a different place but now I've added that as the first step in tryComplete ([code reference](https://github.com/apache/kafka/pull/17539/files#diff-d835cdc01e77905316584ce9e6e21a060cb3d36efa717d4b822b16744e4d713aR151)). As you had also mentioned, this prevents us from making multiple share partition null checks at different places in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816711409


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   That's my queustion with this comment https://github.com/apache/kafka/pull/17539#discussion_r1816703702. Do we ever get the lastFetchOffsetMetadata or we always update with fetchOffsetMetadata?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1817933790


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hi @junrao , i am not sure if I understand the approach completely. IIUC, 
   1. I agree with the common case where during `onComplete` we can update the file position by directly adding the batch size. But, there are cases where we return true from `isMinBytesSatisfied` if `fetchOffsetMetadata` was on a different segment than the `endOffsetMetadata` ([code reference](https://github.com/apache/kafka/pull/17539/files#diff-d835cdc01e77905316584ce9e6e21a060cb3d36efa717d4b822b16744e4d713aR296)). In those case, we don't know the number of bytes that got accumulated in the response. Same goes for the case `fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset`, how do we know whta is the number of bytes to add?
   2. I am assuming that with this approach, we do not need to do any handling during acquisition lock timeout/acknowledgements/release acquired records on session close. Please correct me if I am wrong.
   PS - Thanks a lot for taking out the time to explain me and reviewing this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828824435


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();

Review Comment:
   Yes, there have been comments above where I initialized some variables as null, and it was pointed out that I need to initialize them to avoid null checks at different places, then we just need to do empty checks.. Hence, I've implemented it in this manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828824435


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();

Review Comment:
   Yes, there have been comments above where I initialized some variables as null, and it was pointed out that I need to initialize them to avoid null checks. Hence, I've implemented it in this manner. https://github.com/apache/kafka/pull/17539#discussion_r1819637938



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828836294


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -207,13 +243,155 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
-        topicIdPartitions.forEach(tp -> {
-            SharePartition sharePartition = sharePartitionManager.sharePartition(groupId, tp);
+    private Map<TopicIdPartition, LogReadResult> maybeReadFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        topicPartitionData.forEach((topicIdPartition, partitionData) -> {
+            SharePartition sharePartition = sharePartitions.get(topicIdPartition);
+            if (sharePartition.fetchOffsetMetadata().isEmpty()) {
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, partitionData);
+            }
+        });
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {

Review Comment:
   Now, we reset fetchOffsetMetadata everytime the `endOffset` changes (see the usages of function `updateEndOffsetAndResetFetchOffsetMetadata` in `SharePartition`). So, it can be frequent now that the fetchOffsetMetadata is empty. https://github.com/apache/kafka/pull/17539#discussion_r1824931761



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826562798


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   Hi @junrao, I am not actually too sure if I definitely need this check. Ideally, I don't think there can be any value in `sharePartitions` which can be null, but this [TODO](https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/server/share/SharePartitionManager.java#L629) in `SharePartitionManager` is confusing me, plus there is this JIRA https://issues.apache.org/jira/browse/KAFKA-17510 where we will be refactoring share partition initialization. So, this can act as a safety check for now, and I can remove this in a future PR once the refactor is complete, and we are sure we don't send null share partition. What do you think?
   cc - @apoorvmittal10 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830468964


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -51,21 +55,26 @@ public class DelayedShareFetch extends DelayedOperation {
 
     private static final Logger log = LoggerFactory.getLogger(DelayedShareFetch.class);
 
-    private final ShareFetchData shareFetchData;
+    private final ShareFetchData partitionsToComplete;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsAcquired;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
-            ShareFetchData shareFetchData,
+            ShareFetchData partitionsToComplete,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
-        super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
-        this.shareFetchData = shareFetchData;
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
+        super(partitionsToComplete.fetchParams().maxWaitMs, Option.empty());
+        this.partitionsToComplete = partitionsToComplete;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsAcquired = new LinkedHashMap<>();

Review Comment:
   Similar reason as [above](https://github.com/apache/kafka/pull/17539#discussion_r1830454251), I did it for performance efficiency. We are iterating over `partitionsAcquired` in `releasePartitionLocks`, hence I thought it would be more efficient to use LinkedHashMap over HashMap.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on PR #17539:
URL: https://github.com/apache/kafka/pull/17539#issuecomment-2461751459

   Hi @junrao , I've addressed the comments from the latest review. Please re-review my PR when you get a chance. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806639708


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());

Review Comment:
   I think you are right. I also see only reference of param minBytes in fetchMessages and not in readFromLog. Also the readFromLog says upto maximum in description and nothing about minBytes.
   
   Then the PR change sounds good but I was wondering why do we accept complete fetchParams in readFromLog when we don't utilize something like minBytes there. Not sure if we should have minBytes support in readFromLog itself. Maybe out of scope of this PR.
   
   @junrao can help is with more context.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1807312461


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;

Review Comment:
   I can get rid of it, but then the variable name `replicaManagerFetchDataFromTryComplete` won't make sense if it is getting some values in `onComplete`. I just feel that the code is more readable this way



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;

Review Comment:
   I can get rid of it, but then the variable name `replicaManagerFetchDataFromTryComplete` won't make sense if it is getting some values in `onComplete`. I just feel that the code is more readable this way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806735371


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -86,57 +86,37 @@ public void onComplete() {
         if (shareFetchData.future().isDone())
             return;
 
-        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
-        // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
-            topicPartitionData = acquirablePartitions();
-        // tryComplete invoked forceComplete, so we can use the data from tryComplete.
-        else
-            topicPartitionData = topicPartitionDataFromTryComplete;
-
-        if (topicPartitionData.isEmpty()) {
-            // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
-            return;
+        Map<TopicIdPartition, FetchPartitionData> fetchResponseData;
+        // tryComplete did not invoke forceComplete, so we need to get replica manager response data.
+        if (replicaManagerFetchDataFromTryComplete.isEmpty()) {
+            Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
+            if (topicPartitionData.isEmpty()) {
+                // No locks for share partitions could be acquired, so we complete the request with an empty response.
+                shareFetchData.future().complete(Collections.emptyMap());
+                return;
+            }
+            fetchResponseData = replicaManagerFetchData(topicPartitionData, true);
+        } else {
+            // tryComplete invoked forceComplete, so we can use the replica manager response data from tryComplete.
+            fetchResponseData = replicaManagerFetchDataFromTryComplete;
         }
-        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-                topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
             Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
+                ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager);
             shareFetchData.future().complete(result);
         } catch (Exception e) {
             log.error("Error processing delayed share fetch request", e);
             shareFetchData.future().completeExceptionally(e);
         } finally {
             // Releasing the lock to move ahead with the next request in queue.
-            releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            releasePartitionLocks(shareFetchData.groupId(), fetchResponseData.keySet());

Review Comment:
   for that case, even when the data is not received from replica manager, the fetchResponseData should still have keys as the locked topic partitions and values as empty data, so it should work. Am I wrong in that understanding?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1812833979


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +217,29 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    private boolean isMinBytesCriteriaSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   nit: I hesitate to correct your Greek, but the singular of "criteria" is "criterion". Maybe "isMinBytesSatisfied" would be simpler than worrying about Greek grammar.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1817193502


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   I was thinking about an alternative approach by maintaining `latestFetchOffsetMetadata` every time we update `endOffset`. In the common case, we move `endOffset` to `lastOffset` of an acked batch. The file position in `latestFetchOffsetMetadata` can just be updated by adding the batch size to the file position of the previous `latestFetchOffsetMetadata`. This way, in the common case, we only need to call `readFromLog` once per fetch request, instead of twice (once for getting `latestFetchOffsetMetadata` and another for getting the data) in the current approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816509367


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -55,6 +60,7 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ReplicaManager replicaManager;
 
     private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchPartitionOffsetData> logReadResponseFromTryComplete;

Review Comment:
   Do we need the variable name to have suffix `TryComplete`? I don't find the suffix is any helpful.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();

Review Comment:
   How frequent it is to see missing fetch offset information, mostly at start only right? They why to initialize this variable, can't it be lazily loaded, if required?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;
+                } else if (fetchOffsetMetadata.messageOffset < endOffsetMetadata.messageOffset) {
+                    if (fetchOffsetMetadata.onOlderSegment(endOffsetMetadata)) {
+                        // This can happen when the fetch operation is falling behind the current segment or the partition
+                        // has just rolled a new segment.
+                        log.debug("Satisfying delayed share fetch request for group {}, member {} immediately since it is fetching older " +
+                            "segments of topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                        return true;
+                    } else if (fetchOffsetMetadata.onSameSegment(endOffsetMetadata)) {
+                        // we take the partition fetch size as upper bound when accumulating the bytes.
+                        long bytesAvailable = Math.min(endOffsetMetadata.positionDiff(fetchOffsetMetadata), partitionData.maxBytes);
+                        accumulatedSize += bytesAvailable;
+                    }
+                }
+            }
+            return accumulatedSize >= shareFetchData.fetchParams().minBytes;
+        } catch (Exception e) {
+            // Ideally we should complete the share fetch request's future exceptionally in this case from tryComplete itself.
+            // A function that can be utilized is handleFetchException in an in-flight PR https://github.com/apache/kafka/pull/16842.
+            // Perhaps, once the mentioned PR is merged, I'll change it to better exception handling.
+            log.error("Error processing the minBytes criteria for share fetch request", e);
+            return true;
+        }
+    }
+
+    private Map<TopicIdPartition, FetchPartitionOffsetData> readFromLog(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+            shareFetchData.fetchParams(),
+            CollectionConverters.asScala(
+                topicPartitionData.entrySet().stream().map(entry ->
+                    new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+            ),
+            QuotaFactory.UnboundedQuota$.MODULE$,
+            true);
+
+        Map<TopicIdPartition, FetchPartitionOffsetData> responseData = new HashMap<>();
+        responseLogResult.foreach(tpLogResult -> {
+            TopicIdPartition topicIdPartition = tpLogResult._1();
+            LogReadResult logResult = tpLogResult._2();
+            FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+            responseData.put(topicIdPartition, new FetchPartitionOffsetData(fetchPartitionData, logResult.info().fetchOffsetMetadata));
+            return BoxedUnit.UNIT;
+        });
+
+        log.trace("Data successfully retrieved by replica manager: {}", responseData);
+        return responseData;
+    }
+
+    // Visible for testing.
+    Map<TopicIdPartition, FetchPartitionOffsetData> combineLogReadResponseForOnComplete(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {

Review Comment:
   Why to have method names with such suffix, are they helping?
   
   ```suggestion
       Map<TopicIdPartition, FetchPartitionOffsetData> combineLogReadResponse(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
   ```



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;

Review Comment:
   @adixitconfluent Is it handled?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -214,4 +368,14 @@ private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicId
             sharePartition.releaseFetchLock();
         });
     }
+
+    static final class MaybeUpdateFetchOffsetMetadataResult {

Review Comment:
   Wouldn't `FetchOffsetMetadataUpdateResult` be a better name?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815510878


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   We need to reset `latestFetchOffsetMetadata` every time `endOffset` changes.



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -542,6 +552,8 @@ public ShareAcquiredRecords acquire(
         RecordBatch firstBatch = fetchPartitionData.records.batches().iterator().next();
         lock.writeLock().lock();
         try {
+            // Update the latest fetch offset metadata for any future queries.
+            this.latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   The `acquire` call always comes after `DelayedShareFetch.tryComplete`, which already updates  `latestFetchOffsetMetadata`. So, it seems that we don't need to update `latestFetchOffsetMetadata` in `acquire`?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata() == null) {

Review Comment:
   Perhaps it's clearer to make sharePartition.latestFetchOffsetMetadata() an Optional?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -153,13 +141,23 @@ public boolean tryComplete() {
             shareFetchData.groupId(), shareFetchData.memberId(),
             shareFetchData.partitionMaxBytes().keySet());
 
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty())
-            return forceComplete();
-        log.info("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
-                "topic partitions {}", shareFetchData.groupId(),
-                shareFetchData.memberId(), shareFetchData.partitionMaxBytes().keySet());
+        if (!topicPartitionData.isEmpty()) {
+            maybeUpdateFetchOffsetMetadataForTopicPartitions(topicPartitionData);

Review Comment:
   It's possible that `maybeUpdateFetchOffsetMetadataForTopicPartitions` calls `readFromLog`, which returns enough bytes. In that case, it's more efficient to reuse the fetched result instead of calling `readFromLog` again in `onComplete`.



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata() == null) {
+                if (replicaManagerReadResponseData == null) {
+                    replicaManagerReadResponseData = readFromLog(topicPartitionData);

Review Comment:
   Hmm, we don't want to read all partitions if only one partition's offset metadata is missing, right?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata() == null) {
+                if (replicaManagerReadResponseData == null) {
+                    replicaManagerReadResponseData = readFromLog(topicPartitionData);
+                }
+                FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+                if (fetchPartitionOffsetData == null) {
+                    log.debug("Replica manager read log result {} does not contain topic partition {}",
+                        replicaManagerReadResponseData, topicIdPartition);
+                    continue;
+                }
+                sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            }
+        }
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        AtomicLong accumulatedSize = new AtomicLong(0);

Review Comment:
   Could this just be a `long`?



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;

Review Comment:
   In DelayedFetch, if a partition no longer exists, we complete the operation immediately.



##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -277,6 +279,12 @@ public static RecordState forId(byte id) {
      */
     private final ReplicaManager replicaManager;
 
+    /**
+     * We maintain the latest fetch offset metadata in order to know the last segment position that has been fetched

Review Comment:
   1. Perhaps "We maintain the latest fetch offset metadata to estimate the minBytes requirement more efficiently."? 
   2. Also, could we keep it together with `endOffset` since they are related.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1818144589


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   While this alternative approach is more efficient, it's probably also more complicated. So, it's also ok to just take the current approach to start with. In the current approach, (1) if any call moves `endOffset`, we reset the `latestFetchOffsetMetadata` to Optional.empty(). In `tryComplete`, if `latestFetchOffsetMetadata` is empty, we call `readFromLog` and update `latestFetchOffsetMetadata`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828820632


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Hi @junrao, you're right, I've changed the line to releasePartitionLocks(topicPartitionData.keySet())



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Hi @junrao, you're right, I've changed the line to `releasePartitionLocks(topicPartitionData.keySet())`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828820632


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,51 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {
+            return forceComplete();
+        }
+        Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData = acquirablePartitions();
 
-        if (!topicPartitionDataFromTryComplete.isEmpty()) {
+        try {
+            if (!topicPartitionData.isEmpty()) {
+                // In case, fetch offset metadata doesn't exist for one or more topic partitions, we do a
+                // replicaManager.readFromLog to populate the offset metadata and update the fetch offset metadata for
+                // those topic partitions.
+                Map<TopicIdPartition, LogReadResult> replicaManagerReadResponse = updateFetchOffsetMetadata(maybeReadFromLog(topicPartitionData));
+                if (anyTopicIdPartitionHasLogReadError(replicaManagerReadResponse) || isMinBytesSatisfied(topicPartitionData)) {
+                    partitionsToComplete = topicPartitionData;
+                    partitionsAlreadyFetched = replicaManagerReadResponse;
+                    boolean completedByMe = forceComplete();
+                    // If invocation of forceComplete is not successful, then that means the request is already completed
+                    // hence release the acquired locks.
+                    if (!completedByMe) {
+                        releasePartitionLocks(partitionsToComplete.keySet());
+                        partitionsAlreadyFetched.clear();
+                        partitionsToComplete.clear();
+                    }
+                    return completedByMe;
+                } else {
+                    log.debug("minBytes is not satisfied for the share fetch request for group {}, member {}, " +
+                            "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                        shareFetchData.partitionMaxBytes().keySet());
+                    releasePartitionLocks(topicPartitionData.keySet());
+                }
+            } else {
+                log.trace("Can't acquire records for any partition in the share fetch request for group {}, member {}, " +
+                        "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
+                    shareFetchData.partitionMaxBytes().keySet());
+            }
+            return false;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
             boolean completedByMe = forceComplete();
             // If invocation of forceComplete is not successful, then that means the request is already completed
             // hence release the acquired locks.
             if (!completedByMe) {
-                releasePartitionLocks(shareFetchData.groupId(), topicPartitionDataFromTryComplete.keySet());
+                releasePartitionLocks(partitionsToComplete.keySet());

Review Comment:
   Hi @junrao, its not always a guarantee that `partitionsToComplete` has been set when we hit this exception, but even in that case this line of code shouldn't be a problem since we are are initializing it as an empty map and also we clear the map if the invocation of `forceComplete` is unsuccessful post `isMinBytesSatisfied` check. So, I can add a check that `partitionsToComplete` is not empty, then call this line, but I don't think even this line is a problem. Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1828825732


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -54,18 +59,23 @@ public class DelayedShareFetch extends DelayedOperation {
     private final ShareFetchData shareFetchData;
     private final ReplicaManager replicaManager;
 
-    private Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionDataFromTryComplete;
+    private Map<TopicIdPartition, FetchRequest.PartitionData> partitionsToComplete;
+    private Map<TopicIdPartition, LogReadResult> partitionsAlreadyFetched;
     private final SharePartitionManager sharePartitionManager;
+    private final Map<TopicIdPartition, SharePartition> sharePartitions;
 
     DelayedShareFetch(
             ShareFetchData shareFetchData,
             ReplicaManager replicaManager,
-            SharePartitionManager sharePartitionManager) {
+            SharePartitionManager sharePartitionManager,
+            Map<TopicIdPartition, SharePartition> sharePartitions) {
         super(shareFetchData.fetchParams().maxWaitMs, Option.empty());
         this.shareFetchData = shareFetchData;
         this.replicaManager = replicaManager;
-        this.topicPartitionDataFromTryComplete = new LinkedHashMap<>();
+        this.partitionsToComplete = new LinkedHashMap<>();
+        this.partitionsAlreadyFetched = new LinkedHashMap<>();

Review Comment:
   Same reason as above, we are avoiding any null checks. We just check for empty scenarios by doing this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806725239


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }
+        }
+        // This call is coming from onComplete, hence we return the response data irrespective of whether minBytes is
+        // satisfied or not.
+        if (hasRequestTimedOut)
+            return responseData;

Review Comment:
   Now that I think again,  I should return a map with key as topic partition and value as `fetchPartitionData` object containing 0 records and since we have not been able to satisfy all the fetch request criterias. @junrao your thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806716182


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }

Review Comment:
   in this case, we want to call `ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager)` before we want to release the locks. This part is in `onComplete`, hence we don't release the lock



##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());
+                return BoxedUnit.UNIT;
+            });
+            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+
+            if (accumulatedBytes.get() >= shareFetchData.fetchParams().minBytes)
+                replicaManagerFetchSatisfyingMinBytes = responseData;
+        } catch (Exception e) {
+            log.error("Error processing delayed share fetch request", e);
+        } finally {
+            // The case when we cannot satisfy the share fetch requests because the response has lesser data than minBytes
+            // and the call is coming from tryComplete, hence we want to release partitions lock so that the next
+            // tryComplete/onComplete call complete successfully.
+            if (replicaManagerFetchSatisfyingMinBytes.isEmpty() && !hasRequestTimedOut) {
+                // Releasing the lock to move ahead with the next request in queue.
+                releasePartitionLocks(shareFetchData.groupId(), topicPartitionData.keySet());
+            }

Review Comment:
   in this case, we want to call `ShareFetchUtils.processFetchResponse(shareFetchData, fetchResponseData, sharePartitionManager, replicaManager)` before we want to release the locks. That part is in `onComplete`, hence we don't release the lock



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1817943465


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   @adixitconfluent : Here is the rough idea.
   1. If endOffset advances forward, we incrementally update its file position by the size of batches going forwarded.
   2. The tricky thing is how the offset metadata picks up a new segment being rolled. As we increase the file position, endOffset will eventually reach the baseOffset of the next segment. This means that the next fetch request will be satisfied immediately since the HWM is on a different segment. When we acquire the data (for batches at the beginning of the segment), we can check if the offset metadata in the fetch data has the same offset as endOffset but on a different segment. If so, we update the offset segment of endOffset.
   3. If the endOffset goes backward (due to timeout/acknowledgements/release) or endOffset is being initialized for the first time, we just call readFromLog to get the offset metadata.
   4. `tryComplete` will have the same logic to deal with the uncommon cases where the offset metadata is not available or the offset metadata is on the same segment. The only difference is that it won't update latestFetchOffsetMetadata any more since the update happens when endOffset changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao merged PR #17539:
URL: https://github.com/apache/kafka/pull/17539


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1830461947


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -80,64 +89,56 @@ public void onExpiration() {
     @Override
     public void onComplete() {
         log.trace("Completing the delayed share fetch request for group {}, member {}, "
-            + "topic partitions {}", shareFetchData.groupId(), shareFetchData.memberId(),
-            topicPartitionDataFromTryComplete.keySet());
+            + "topic partitions {}", partitionsToComplete.groupId(), partitionsToComplete.memberId(),
+            partitionsAcquired.keySet());
 
-        if (shareFetchData.future().isDone())
+        if (partitionsToComplete.future().isDone())
             return;
 
         Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData;
         // tryComplete did not invoke forceComplete, so we need to check if we have any partitions to fetch.
-        if (topicPartitionDataFromTryComplete.isEmpty())
+        if (partitionsAcquired.isEmpty())
             topicPartitionData = acquirablePartitions();
         // tryComplete invoked forceComplete, so we can use the data from tryComplete.
         else
-            topicPartitionData = topicPartitionDataFromTryComplete;
+            topicPartitionData = partitionsAcquired;
 
         if (topicPartitionData.isEmpty()) {
             // No locks for share partitions could be acquired, so we complete the request with an empty response.
-            shareFetchData.future().complete(Collections.emptyMap());
+            partitionsToComplete.future().complete(Collections.emptyMap());
             return;
         }
         log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}",
-            topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
+            topicPartitionData, partitionsToComplete.groupId(), partitionsToComplete.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UNBOUNDED_QUOTA,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
-            Map<TopicIdPartition, ShareFetchResponseData.PartitionData> result =
-                ShareFetchUtils.processFetchResponse(shareFetchData, responseData, sharePartitionManager, replicaManager);
-            shareFetchData.future().complete(result);
+            Map<TopicIdPartition, LogReadResult> responseData;
+            if (partitionsAlreadyFetched.isEmpty())
+                responseData = readFromLog(topicPartitionData);
+            else
+                // There shouldn't be a case when we have a partitionsAlreadyFetched value here and this variable is getting
+                // updated in a different tryComplete thread.

Review Comment:
   Hi @junrao , I've created a ticket https://issues.apache.org/jira/browse/KAFKA-17948 to track this issue and if it fine to you, I would prefer to address the issue in a future PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1826562798


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -146,18 +149,47 @@ public void onComplete() {
      */
     @Override
     public boolean tryComplete() {
-        topicPartitionDataFromTryComplete = acquirablePartitions();
+        if (anySharePartitionNoLongerExists()) {

Review Comment:
   Hi @junrao, I am not actually too sure if I definitely need this check. Ideally, I don't think there can be any value in `sharePartitions` which can be null, but this [TODO](https://github.com/apache/kafka/blob/trunk/core/src/main/java/kafka/server/share/SharePartitionManager.java#L629) in `SharePartitionManager` is confusing me, plus there is this JIRA https://issues.apache.org/jira/browse/KAFKA-17510 where we will be refactoring share partition initialization. So, this can act as a safety check for now, and I can remove this in a future PR once the refactor is complete, and we are sure we don't send null share partitions. What do you think?
   cc - @apoorvmittal10 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816830605


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -103,25 +109,11 @@ public void onComplete() {
                 topicPartitionData, shareFetchData.groupId(), shareFetchData.fetchParams());
 
         try {
-            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
-                shareFetchData.fetchParams(),
-                CollectionConverters.asScala(
-                    topicPartitionData.entrySet().stream().map(entry ->
-                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
-                ),
-                QuotaFactory.UnboundedQuota$.MODULE$,
-                true);
-
-            Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
-            responseLogResult.foreach(tpLogResult -> {
-                TopicIdPartition topicIdPartition = tpLogResult._1();
-                LogReadResult logResult = tpLogResult._2();
-                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
-                responseData.put(topicIdPartition, fetchPartitionData);
-                return BoxedUnit.UNIT;
-            });
-
-            log.trace("Data successfully retrieved by replica manager: {}", responseData);
+            Map<TopicIdPartition, FetchPartitionOffsetData> responseData;
+            if (logReadResponseFromTryComplete == null || logReadResponseFromTryComplete.isEmpty())

Review Comment:
   you're right, I don't need it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816704806


##########
share/src/main/java/org/apache/kafka/server/share/fetch/FetchPartitionOffsetData.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.server.share.fetch;
+
+import org.apache.kafka.server.storage.log.FetchPartitionData;
+import org.apache.kafka.storage.internals.log.LogOffsetMetadata;
+
+public class FetchPartitionOffsetData {

Review Comment:
   Class comments please.



##########
share/src/main/java/org/apache/kafka/server/share/fetch/FetchPartitionOffsetData.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.server.share.fetch;
+
+import org.apache.kafka.server.storage.log.FetchPartitionData;
+import org.apache.kafka.storage.internals.log.LogOffsetMetadata;
+
+public class FetchPartitionOffsetData {
+    private final FetchPartitionData fetchPartitionData;
+    private final LogOffsetMetadata logOffsetMetadata;
+
+    public FetchPartitionOffsetData(FetchPartitionData fetchPartitionData, LogOffsetMetadata logOffsetMetadata) {
+        this.fetchPartitionData = fetchPartitionData;
+        this.logOffsetMetadata = logOffsetMetadata;
+    }
+
+    public FetchPartitionData fetchPartitionData() {
+        return fetchPartitionData;
+    }
+
+    public LogOffsetMetadata logOffsetMetadata() {
+        return logOffsetMetadata;
+    }
+}

Review Comment:
   Empty line break please.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816813489


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }

Review Comment:
   So this suggestion is not apt as we need the refreshed information https://github.com/apache/kafka/pull/17539#discussion_r1816800887



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "adixitconfluent (via GitHub)" <gi...@apache.org>.
adixitconfluent commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1816851362


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +211,154 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private MaybeUpdateFetchOffsetMetadataResult maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        boolean isFetchOffsetMetadataUpdated = false;
+        Map<TopicIdPartition, FetchRequest.PartitionData> missingFetchOffsetMetadataTopicPartitions = new LinkedHashMap<>();
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata().isEmpty())
+                missingFetchOffsetMetadataTopicPartitions.put(topicIdPartition, entry.getValue());
+        }
+
+        if (missingFetchOffsetMetadataTopicPartitions.isEmpty()) {
+            return new MaybeUpdateFetchOffsetMetadataResult(false, null);
+        }
+        // We fetch data from replica manager corresponding to the topic partitions that have missing fetch offset metadata.
+        replicaManagerReadResponseData = readFromLog(missingFetchOffsetMetadataTopicPartitions);
+
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : missingFetchOffsetMetadataTopicPartitions.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.debug("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            FetchPartitionOffsetData fetchPartitionOffsetData = replicaManagerReadResponseData.get(topicIdPartition);
+            if (fetchPartitionOffsetData == null) {
+                log.debug("Replica manager read log result {} does not contain topic partition {}",
+                    replicaManagerReadResponseData, topicIdPartition);
+                continue;
+            }
+            sharePartition.updateLatestFetchOffsetMetadata(fetchPartitionOffsetData.logOffsetMetadata());
+            isFetchOffsetMetadataUpdated = true;
+        }
+        return new MaybeUpdateFetchOffsetMetadataResult(isFetchOffsetMetadataUpdated, replicaManagerReadResponseData);
+    }
+
+    private boolean isMinBytesSatisfied(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        long accumulatedSize = 0;
+        try {
+            for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+                TopicIdPartition topicIdPartition = entry.getKey();
+                FetchRequest.PartitionData partitionData = entry.getValue();
+                Partition partition = replicaManager.getPartitionOrException(topicIdPartition.topicPartition());
+                LogOffsetSnapshot offsetSnapshot = partition.fetchOffsetSnapshot(Optional.empty(), true);
+                // The FetchIsolation type that we use for share fetch is FetchIsolation.HIGH_WATERMARK. In the future, we can
+                // extend it other FetchIsolation types.
+                FetchIsolation isolationType = shareFetchData.fetchParams().isolation;
+                LogOffsetMetadata endOffsetMetadata;
+                if (isolationType == FetchIsolation.LOG_END)
+                    endOffsetMetadata = offsetSnapshot.logEndOffset;
+                else if (isolationType == FetchIsolation.HIGH_WATERMARK)
+                    endOffsetMetadata = offsetSnapshot.highWatermark;
+                else
+                    endOffsetMetadata = offsetSnapshot.lastStableOffset;
+
+                if (endOffsetMetadata == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+
+                SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+                if (sharePartition == null) {
+                    return true;
+                }
+
+                Optional<LogOffsetMetadata> optionalFetchOffsetMetadata = sharePartition.latestFetchOffsetMetadata();
+                if (optionalFetchOffsetMetadata.isEmpty() || optionalFetchOffsetMetadata.get() == LogOffsetMetadata.UNKNOWN_OFFSET_METADATA)
+                    continue;
+                LogOffsetMetadata fetchOffsetMetadata = optionalFetchOffsetMetadata.get();
+
+                if (fetchOffsetMetadata.messageOffset > endOffsetMetadata.messageOffset) {
+                    log.debug("Satisfying delayed share fetch request for group {}, member {} since it is fetching later segments of " +
+                        "topicIdPartition {}", shareFetchData.groupId(), shareFetchData.memberId(), topicIdPartition);
+                    return true;

Review Comment:
   It might not be true ever in case we are using `FetchIsolation.HIGH_WATERMARK`, but I think it can be true in case we use `FetchIsolation.LOG_END` and `FetchIsolation.TXN_COMMITTED` which might be used in the future in share fetch requests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1806596264


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,6 +187,58 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
+    /**
+     * Prepare partitions fetch data structure for acquirable partitions in the share fetch request satisfying minBytes criteria.
+     */
+    Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchData(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData,
+                                                                      boolean hasRequestTimedOut) {
+        log.trace("Fetchable share partitions data: {} with groupId: {} fetch params: {}", topicPartitionData,
+            shareFetchData.groupId(), shareFetchData.fetchParams());
+        Map<TopicIdPartition, FetchPartitionData> replicaManagerFetchSatisfyingMinBytes = new HashMap<>();
+        Map<TopicIdPartition, FetchPartitionData> responseData = new HashMap<>();
+        try {
+            Seq<Tuple2<TopicIdPartition, LogReadResult>> responseLogResult = replicaManager.readFromLog(
+                shareFetchData.fetchParams(),
+                CollectionConverters.asScala(
+                    topicPartitionData.entrySet().stream().map(entry ->
+                        new Tuple2<>(entry.getKey(), entry.getValue())).collect(Collectors.toList())
+                ),
+                QuotaFactory.UnboundedQuota$.MODULE$,
+                true);
+
+            AtomicInteger accumulatedBytes = new AtomicInteger(0);
+
+            responseLogResult.foreach(tpLogResult -> {
+                TopicIdPartition topicIdPartition = tpLogResult._1();
+                LogReadResult logResult = tpLogResult._2();
+                FetchPartitionData fetchPartitionData = logResult.toFetchPartitionData(false);
+                responseData.put(topicIdPartition, fetchPartitionData);
+                accumulatedBytes.addAndGet(logResult.info().records.sizeInBytes());

Review Comment:
   Question for my understanding: Do we need to this calculation? As I can see fetch params already has minBytes in request to replica manager hence isn't the response from replica manager should be empty if minBytes criteria is not satisfied?
   
   So the question arise that how do we differentiate between empty reponse from replica manager log read, if that's beacus of min bytes or there is no data in the log? In either case we should continue holding the request in purgatory? Wdyt? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815741519


##########
core/src/main/java/kafka/server/share/DelayedShareFetch.java:
##########
@@ -204,7 +202,117 @@ Map<TopicIdPartition, FetchRequest.PartitionData> acquirablePartitions() {
         return topicPartitionData;
     }
 
-    private void releasePartitionLocks(String groupId, Set<TopicIdPartition> topicIdPartitions) {
+    // In case, fetch offset metadata doesn't exist for any topic partition in the list of topic partitions, we do a
+    // replicaManager.readFromLog to populate the offset metadata.
+    private void maybeUpdateFetchOffsetMetadataForTopicPartitions(Map<TopicIdPartition, FetchRequest.PartitionData> topicPartitionData) {
+        Map<TopicIdPartition, FetchPartitionOffsetData> replicaManagerReadResponseData = null;
+        for (Map.Entry<TopicIdPartition, FetchRequest.PartitionData> entry : topicPartitionData.entrySet()) {
+            TopicIdPartition topicIdPartition = entry.getKey();
+            SharePartition sharePartition = sharePartitionManager.sharePartition(shareFetchData.groupId(), topicIdPartition);
+            if (sharePartition == null) {
+                log.error("Encountered null share partition for groupId={}, topicIdPartition={}. Skipping it.", shareFetchData.groupId(), topicIdPartition);
+                continue;
+            }
+            if (sharePartition.latestFetchOffsetMetadata() == null) {
+                if (replicaManagerReadResponseData == null) {
+                    replicaManagerReadResponseData = readFromLog(topicPartitionData);

Review Comment:
   Maybe we can first make a pass to collect all partitions missing offset metadata and then make a single `readFromLog` with those partitions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-17743: Add minBytes implementation to DelayedShareFetch [kafka]

Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #17539:
URL: https://github.com/apache/kafka/pull/17539#discussion_r1815731593


##########
core/src/main/java/kafka/server/share/SharePartition.java:
##########
@@ -1525,6 +1537,24 @@ private Optional<Throwable> acknowledgeCompleteBatch(
         return Optional.empty();
     }
 
+    protected void updateLatestFetchOffsetMetadata(LogOffsetMetadata fetchOffsetMetadata) {
+        lock.writeLock().lock();
+        try {
+            latestFetchOffsetMetadata = fetchOffsetMetadata;

Review Comment:
   Hmm, if a previously acquired batch times out, SharePartition will change endOffset. This means that the cached latestFetchOffsetMetadata no longer matches the next fetch offset, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org