[jira] [Commented] (KAFKA-249) Separate out Kafka mirroring into a stand-alone app

["Jun Rao (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246497#comment-13246497 ] 

Jun Rao commented on KAFKA-249:
-------------------------------

v2 looks much better now. Some additional comments:

21. Should isTopicAllowed be part of TopicFilterSpec, especially if we want to extend it in the future? If so, we don't need TopicFilter.
22. In this patch, I suggest that we only put message and topic in MessageAndMetadata. We can have a separate jira on how to expose offsets to the consumer. There, we need to discuss how a consume can rewind the consumption using the offset returned.
23. It's probably better to rename KafkaMessageAndMetadataStream to KafkaStream.
24. ZookeeperConsumerConnector:
24.1 reinitializeConsumer:  I think it will make the code easier to understand if we explicitly define the type of val consumerThreadIdsPerTopic, topicThreadIds and threadQueueStreamPairs. It would be also very useful to explicitly define the return type of consumerThreadIdsPerTopic.flatten.
24.2 reinitializeConsumer: This method is called every time a new topic is discovered. It feels strange that we have to register the consumer here. Ideally, each consumer is registered exactly once. Also, it seems that each time this method is called, we only add new entries to loadBalancerListener.kafkaMessageAndMetadataStreams. Shouldn't we clear this map first so that deleted topics can be removed?

25. ByteBufferMessageSet: It's not clear to me if the iterator of ByteBufferMessageSet should return MessageAndMetadata. This is because ByteBufferMessageSet itself doesn't know all the metadata, such as topic and partition. So, it seems the iterator of this class should probably remain MessageAndOffset. MessageAndMetadata is only used for the client api.

26. MirrorMaker: The shutdown hook should close producer.

                
> Separate out Kafka mirroring into a stand-alone app
> ---------------------------------------------------
>
>                 Key: KAFKA-249
>                 URL: https://issues.apache.org/jira/browse/KAFKA-249
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>             Fix For: 0.7.1
>
>         Attachments: KAFKA-249.v1.patch, KAFKA-249.v2.patch, KAFKA-249.v3.patch
>
>
> I would like to discuss on this jira, the feasibility/benefits of separating
> out Kafka's mirroring feature from the broker into a stand-alone app, as it
> currently has a couple of limitations and issues.
> For example, we recently had to deal with Kafka mirrors that were in fact
> idle due to the fact that mirror threads were not created at start-up due to
> a rebalancing exception, but the Kafka broker itself did not shutdown. This
> has since been fixed, but is indicative of (avoidable) problems in embedding
> non-broker specific features in the broker.
> Logically, it seems to make sense to separate it out to achieve better
> division of labor.  Furthermore, enhancements to mirroring may be less
> clunky to implement and use with a stand-alone app.  For example to support
> custom partitioning on the target cluster, or to mirror from multiple
> clusters we would probably need to be able to pass in multiple embedded
> consumer/embedded producer configs, which would be less ugly if the
> mirroring process were a stand-alone app.  Also, if we break it out, it
> would be convenient to use as a "consumption engine" for the console
> consumer which will make it easier to add on features such as wildcards in
> topic consumption, since it contains a ZooKeeper topic discovery component.
> Any suggestions and/or objections to this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira