[jira] [Commented] (KAFKA-7146) Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539129#comment-16539129 ] 

ASF GitHub Bot commented on KAFKA-7146:
---------------------------------------

shunge opened a new pull request #5353: KAFKA-7146: Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
URL: https://github.com/apache/kafka/pull/5353
 
 
   This is a subtask from KAFKA-6788 (https://issues.apache.org/jira/browse/KAFKA-6788)
   
   Basically the idea is to use thread-safe collections like ConcurrentLinkedQueue and ConcurrentHashMap to save to result of each "findCoordinator" response. In the map, we will have the coordinator node Id as the key, and the value will be the collection of all the belonging groupIds. 
   
   When all the groupId processed (either successful or completed exceptionally), then we will iterate through the map and group the requests by coordinators.
   
   This PR is mainly for discussion, there are a couple concerns:
   
   1. Code logic: Is there any thing wrong with the logic of this solution?
   2. Concurrency issues: can these collections guarantee the thread-safety?
   3. Memory issue: using too many collections too store information, should cut down the memory usage.
   4. Efficiency: is this more efficient than the implementation before?
   5. Naming issues (I already noticed a typo).
   6. More unit tests.
   
   I am open to all kind of suggestions and feedbacks, and will make several other commits after I receive opinion. 
   
   P.S. Is a KIP needed?
   
   @guozhangwang @cmccabe Thank you!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Grouping consumer requests per consumer coordinator in admin client in describeConsumerGroups
> ---------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-7146
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7146
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Yishun Guan
>            Assignee: Yishun Guan
>            Priority: Critical
>
> Subtask of KAFKA-6788. Group consumer requests for describeConsumerGroups().
>  
> According to KAFKA-6788:
>  
> "In KafkaAdminClient, for some requests like describeGroup and deleteGroup, we will first try to get the coordinator for each requested group id, and then send the corresponding request for that group id. However, different group ids could be hosted on the same coordinator, and these requests do support multi group ids be sent within the same request. So we can consider optimize it by grouping the requests per coordinator destination."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)