[jira] [Commented] (KAFKA-473) Use getMetadata Api in ZookeeperConsumerConnector

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461894#comment-13461894 ] 

Jun Rao commented on KAFKA-473:
-------------------------------

Thanks for patch v1. Looks good overall. Some comments:

1. ProducerPool.createSyncProducer(): It's probably better to use the input parameter config an Option than allowing it to be null.

2. Utils.getBrokerPartitionInfo(): We should probably rename this to something like getTopicMetadata() and change log messages with "broker partition metadata" to "topic metadata".

3. It's probably worthwhile to use the getTopicMetadata utility to replace getPartitionsForTopics in ZookeeperConsumerConnector.rebalance(). This saves the reading of all partitions. We still need to do 1 delete and 1 write to ZK for each partition that a consumer owns. However, that's only a subset of the partitions.
                
> Use getMetadata Api in ZookeeperConsumerConnector
> -------------------------------------------------
>
>                 Key: KAFKA-473
>                 URL: https://issues.apache.org/jira/browse/KAFKA-473
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>            Reporter: Jun Rao
>            Assignee: Yang Ye
>            Priority: Blocker
>              Labels: newbie, optimization
>             Fix For: 0.8
>
>         Attachments: kafka_473_v1.diff
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Currently, ZookeeperConsumerConnector gets topic metadata from ZK directly. It's better to use the getMetadata Api since it's batched. This is especially helpful if the consumer client is in a different data center.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira