Re: Can Mirroring Preserve Every Topic's Partition?

[Neha Narkhede <ne...@gmail.com>]

As Guozhang mentioned, that will allow exact partitioning but note that
this still doesn't ensure identical partitions since your target cluster
can have duplicates depending on whether the producer retried sending data
or not.

On Wed, Nov 26, 2014 at 10:33 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hello Alex,
>
> This can be done by doing some tweaks in the MM code (with the 0.8.2 new
> producer).
>
> 1. Set-up your MM to have the total # of producers equal to the #. of
> partitions in source / target cluster.
>
> 2. When the consumer of the MM gets a message, put the message to the
> producer's queue based on its partition id; i.e. if the partition id is n,
> put to n's producer queue.
>
> 3. When producer sends the data, specify the partition id; so each producer
> will only send to a single partition.
>
> Guozhang
>
>
> On Tue, Nov 25, 2014 at 8:19 PM, Alex Melville <am...@g.hmc.edu>
> wrote:
>
> > Howdy friends,
> >
> >
> > I'd like to mirror the topics on several clusters to a central cluster,
> and
> > I'm looking at using the default Mirrormaker to do so. I've already done
> > some basic testing on the Mirrormaker found here:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
> >
> > and managed to successfully copy a topic's partitions on a source cluster
> > to a topic on a target cluster. So I'm able to mirror correctly. However
> > for my particular use case I need to ensure that when I copy a topic's
> > partitions from source cluster to target cluster, a partition created on
> > the target cluster contains data in the exact same order as the data on
> the
> > corresponding partition on the source cluster.
> >
> > I'm thinking of writing a Simple Consumer so I can manually compare the
> > events in a source cluster's partition with the corresponding partition
> on
> > the target cluster, but I'm not 100% sure if I'll be able to verify my
> > guarantee if I do it this way. Can anyone here verify that partitions
> > copied over to the target cluster by the default Mirrormaker are an exact
> > copy of those on the source cluster?
> >
> >
> > Thanks in advance,
> >
> > Alex Melville
> >
>
>
>
> --
> -- Guozhang
>