[GitHub] [pinot] navina commented on issue #11001: optimize kafka `computePartitionGroupMetadata` with KafkaAdminClient

["navina (via GitHub)" <gi...@apache.org>]

navina commented on issue #11001:
URL: https://github.com/apache/pinot/issues/11001#issuecomment-1613195924

   @dang-stripe The number of consumers created in a single server is same as the number of partitions being consumed on the server. Additionally, it creates multiple short-lived consumer instances to fetch topic metadata, which is what you see in `_partitionMetadataProvider` . 
   This is purely due to how the code is organized in the kafka consumer plugin in Pinot (`KafkaStreamMetadataProvider` inherits from `KafkaPartitionLevelConnectionHandler`). I also mention it in this issue - https://github.com/apache/pinot/issues/10014#issue-1505401315 (see description Note)
   
   >  i'm wondering if there are any blockers to using the KafkaAdminClient's listOffsets call to achieve the same thing:
   
   I don't think there are any. `KafkaAdminClient` didn't exist or was not stable when Pinot's kafka consumer was written. We should replace it avoid all the short-lived consumers. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org