You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "dajac (via GitHub)" <gi...@apache.org> on 2024/03/04 10:30:26 UTC

[PR] MINOR: Add read/write all operation [kafka]

dajac opened a new pull request, #15462:
URL: https://github.com/apache/kafka/pull/15462

   There are a few cases in the group coordinator service where we want to read from or write to each of the known coordinators (each of __consumer_offsets partitions). The current implementation needs to get the list of the known coordinators then schedules the operation and finally aggregate the results. This patch is an attempt to streamline this by adding multi read/write to the runtime.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1513587296


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -498,29 +497,17 @@ public CompletableFuture<ListGroupsResponseData> listGroups(
             );
         }
 
-        final Set<TopicPartition> existingPartitionSet = runtime.partitions();
-
-        if (existingPartitionSet.isEmpty()) {
-            return CompletableFuture.completedFuture(new ListGroupsResponseData());
-        }
-
-        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures =
-            new ArrayList<>();
-
-        for (TopicPartition tp : existingPartitionSet) {
-            futures.add(runtime.scheduleReadOperation(
-                "list-groups",
-                tp,
-                (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
-            ).exceptionally(exception -> {
-                exception = Errors.maybeUnwrapException(exception);
-                if (exception instanceof NotCoordinatorException) {
-                    return Collections.emptyList();
-                } else {
-                    throw new CompletionException(exception);
-                }
-            }));
-        }
+        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures = runtime.scheduleReadAllOperation(
+            "list-groups",
+            (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
+        ).stream().map(future -> future.exceptionally(exception -> {

Review Comment:
   how about adding the exception function to `scheduleReadAllOperation`? It brings two benefits:
   
   1. simplify the code from callers 
   2. avoid creating list repeatedly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1516221969


##########
server-common/src/main/java/org/apache/kafka/server/util/FutureUtils.java:
##########
@@ -125,4 +127,21 @@ public static <T> CompletableFuture<T> combineFutures(
             return res;
         });
     }
+
+    /**
+     * Applies the given exception handler to all the futures provided in the list
+     * and returns a new list of futures.
+     *
+     * @param futures   A list of futures.
+     * @param fn        A function taking an exception to handle it.
+     * @return A list of futures.
+     */
+    public static <T> List<CompletableFuture<T>> mapExceptionally(

Review Comment:
   That makes sense to me. We can leave some rooms for refactor if there will be more use cases in the future. Let ship it instead of being over-engineering :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1515976925


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -498,29 +497,17 @@ public CompletableFuture<ListGroupsResponseData> listGroups(
             );
         }
 
-        final Set<TopicPartition> existingPartitionSet = runtime.partitions();
-
-        if (existingPartitionSet.isEmpty()) {
-            return CompletableFuture.completedFuture(new ListGroupsResponseData());
-        }
-
-        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures =
-            new ArrayList<>();
-
-        for (TopicPartition tp : existingPartitionSet) {
-            futures.add(runtime.scheduleReadOperation(
-                "list-groups",
-                tp,
-                (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
-            ).exceptionally(exception -> {
-                exception = Errors.maybeUnwrapException(exception);
-                if (exception instanceof NotCoordinatorException) {
-                    return Collections.emptyList();
-                } else {
-                    throw new CompletionException(exception);
-                }
-            }));
-        }
+        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures = runtime.scheduleReadAllOperation(
+            "list-groups",
+            (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
+        ).stream().map(future -> future.exceptionally(exception -> {

Review Comment:
   I just pushed a commit with the `mapExceptionally` construct that I was thinking about. Please let me know what you think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #15462:
URL: https://github.com/apache/kafka/pull/15462#issuecomment-1976262999

   @OmniaGM Could you take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1516215271


##########
server-common/src/main/java/org/apache/kafka/server/util/FutureUtils.java:
##########
@@ -125,4 +127,21 @@ public static <T> CompletableFuture<T> combineFutures(
             return res;
         });
     }
+
+    /**
+     * Applies the given exception handler to all the futures provided in the list
+     * and returns a new list of futures.
+     *
+     * @param futures   A list of futures.
+     * @param fn        A function taking an exception to handle it.
+     * @return A list of futures.
+     */
+    public static <T> List<CompletableFuture<T>> mapExceptionally(

Review Comment:
   Interesting idea. I would prefer to keep `combineFutures` simple. `mapExceptionally` and `combineFutures` are two different things in my opinion. I also envision that we could have a need for something different that `exceptionally` in the future. If we go on that path, it means that we would need to overload `combineFutures` to support different ones and it will get messy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1514944319


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -498,29 +497,17 @@ public CompletableFuture<ListGroupsResponseData> listGroups(
             );
         }
 
-        final Set<TopicPartition> existingPartitionSet = runtime.partitions();
-
-        if (existingPartitionSet.isEmpty()) {
-            return CompletableFuture.completedFuture(new ListGroupsResponseData());
-        }
-
-        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures =
-            new ArrayList<>();
-
-        for (TopicPartition tp : existingPartitionSet) {
-            futures.add(runtime.scheduleReadOperation(
-                "list-groups",
-                tp,
-                (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
-            ).exceptionally(exception -> {
-                exception = Errors.maybeUnwrapException(exception);
-                if (exception instanceof NotCoordinatorException) {
-                    return Collections.emptyList();
-                } else {
-                    throw new CompletionException(exception);
-                }
-            }));
-        }
+        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures = runtime.scheduleReadAllOperation(
+            "list-groups",
+            (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
+        ).stream().map(future -> future.exceptionally(exception -> {

Review Comment:
   > adding an helper like mapExceptionally to add an exception handler on a list of futures
   
   Do you mean to enhance `FutureUtils#combineFutures` to take one more parameter? Current PR is good to me but I'd like to see more design  (`mapExceptionally`) before merging :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac merged PR #15462:
URL: https://github.com/apache/kafka/pull/15462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1514893077


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupCoordinatorService.java:
##########
@@ -498,29 +497,17 @@ public CompletableFuture<ListGroupsResponseData> listGroups(
             );
         }
 
-        final Set<TopicPartition> existingPartitionSet = runtime.partitions();
-
-        if (existingPartitionSet.isEmpty()) {
-            return CompletableFuture.completedFuture(new ListGroupsResponseData());
-        }
-
-        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures =
-            new ArrayList<>();
-
-        for (TopicPartition tp : existingPartitionSet) {
-            futures.add(runtime.scheduleReadOperation(
-                "list-groups",
-                tp,
-                (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
-            ).exceptionally(exception -> {
-                exception = Errors.maybeUnwrapException(exception);
-                if (exception instanceof NotCoordinatorException) {
-                    return Collections.emptyList();
-                } else {
-                    throw new CompletionException(exception);
-                }
-            }));
-        }
+        final List<CompletableFuture<List<ListGroupsResponseData.ListedGroup>>> futures = runtime.scheduleReadAllOperation(
+            "list-groups",
+            (coordinator, lastCommittedOffset) -> coordinator.listGroups(request.statesFilter(), request.typesFilter(), lastCommittedOffset)
+        ).stream().map(future -> future.exceptionally(exception -> {

Review Comment:
   I actually considered this but eventually rejected it because those new methods could also be used with a different handling than `exceptionally`. The separation of concerns would not be too good in my opinion. I was also considering adding an helper like `mapExceptionally` to add an exception handler on a list of futures. Do you think that it could also help?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #15462:
URL: https://github.com/apache/kafka/pull/15462#discussion_r1516205356


##########
server-common/src/main/java/org/apache/kafka/server/util/FutureUtils.java:
##########
@@ -125,4 +127,21 @@ public static <T> CompletableFuture<T> combineFutures(
             return res;
         });
     }
+
+    /**
+     * Applies the given exception handler to all the futures provided in the list
+     * and returns a new list of futures.
+     *
+     * @param futures   A list of futures.
+     * @param fn        A function taking an exception to handle it.
+     * @return A list of futures.
+     */
+    public static <T> List<CompletableFuture<T>> mapExceptionally(

Review Comment:
   This functionality is good idea. Maybe we can merge this it into `combineFutures` to make `combineFutures` to generate array with exception handler. For example:
   ```java
     public static <T> CompletableFuture<T> combineFutures(
         List<CompletableFuture<T>> futures,
         Supplier<T> init,
         BiConsumer<T, T> add
     ) {
       return combineFutures(futures, init, add, null);
     }
   
     public static <T> CompletableFuture<T> combineFutures(
         List<CompletableFuture<T>> futures,
         Supplier<T> init,
         BiConsumer<T, T> add,
         Function<Throwable, ? extends T> fn
     ) {
       final CompletableFuture<Void> allFutures = fn == null ? CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
         : CompletableFuture.allOf(futures.stream().map(f -> f.exceptionally(fn)).toArray(CompletableFuture[]::new));
       return allFutures.thenApply(v -> {
         final T res = init.get();
         futures.forEach(future -> add.accept(res, future.join()));
         return res;
       });
     }
   ```
   
   The benefit is that we save a collection re-creation, and also `combineFutures` gets more functionality with acceptable complex ( 4 parameters )
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Add read/write all operation [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on PR #15462:
URL: https://github.com/apache/kafka/pull/15462#issuecomment-1979123455

   @chia7712 Would you be interested in reviewing this one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org