[jira] [Updated] (KAFKA-249) Separate out Kafka mirroring into a stand-alone app

["Joel Koshy (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joel Koshy updated KAFKA-249:
-----------------------------

    Attachment: KAFKA-249.v3.patch

This patch addresses the points raised in the review for v2.

I'm terrible at naming methods, so let me know if you have a better name for reinitializeConsumer.

Also, Jay had brought up some good points wrt the API (https://cwiki.apache.org/confluence/display/KAFKA/Consumer+API+changes). I think it would be good to get this reviewed for any other issues even as that conversation goes on.

                
> Separate out Kafka mirroring into a stand-alone app
> ---------------------------------------------------
>
>                 Key: KAFKA-249
>                 URL: https://issues.apache.org/jira/browse/KAFKA-249
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>             Fix For: 0.7.1
>
>         Attachments: KAFKA-249.v1.patch, KAFKA-249.v2.patch, KAFKA-249.v3.patch
>
>
> I would like to discuss on this jira, the feasibility/benefits of separating
> out Kafka's mirroring feature from the broker into a stand-alone app, as it
> currently has a couple of limitations and issues.
> For example, we recently had to deal with Kafka mirrors that were in fact
> idle due to the fact that mirror threads were not created at start-up due to
> a rebalancing exception, but the Kafka broker itself did not shutdown. This
> has since been fixed, but is indicative of (avoidable) problems in embedding
> non-broker specific features in the broker.
> Logically, it seems to make sense to separate it out to achieve better
> division of labor.  Furthermore, enhancements to mirroring may be less
> clunky to implement and use with a stand-alone app.  For example to support
> custom partitioning on the target cluster, or to mirror from multiple
> clusters we would probably need to be able to pass in multiple embedded
> consumer/embedded producer configs, which would be less ugly if the
> mirroring process were a stand-alone app.  Also, if we break it out, it
> would be convenient to use as a "consumption engine" for the console
> consumer which will make it easier to add on features such as wildcards in
> topic consumption, since it contains a ZooKeeper topic discovery component.
> Any suggestions and/or objections to this?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira