You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Madhukar Bharti <bh...@gmail.com> on 2015/01/20 07:51:35 UTC

How to setup inter DC replication in Kafka 0.8.1.1

Hi,

I want to setup inter DC replication b/w Kafka clusters. Is there any
inbuilt tool to do this?

I already have tried MirrorMaker tool but the problem is, if MM killed then
some messages get duplicated. I don't want to duplicate the messages.

Please suggest a way to do this. Please share your experience how you have
setup this.


Thanks in Advance,
Madhukar

Re: How to setup inter DC replication in Kafka 0.8.1.1

Posted by Guozhang Wang <wa...@gmail.com>.

For now (0.8.2) if you have a hard crash then you will likely have
duplicates or even data loss unfortunately, I do not have a good solution
to this on top of my head.

On Wed, Jan 21, 2015 at 10:14 PM, Madhukar Bharti <bh...@gmail.com>
wrote:

> Thanks Guozhang for your reply.
>
> Checked the details as mentioned above. Issue KAFKA-1650 has been fixed but
> it will come with 0.8.3. :-(
>
> By "MM Killed", it is hard-failed(kill -9 )/crashed due to system reboot or
> any other reason. In that case some of the messages get duplicated.
>
> Can you please suggest a way to do inter DC replication for time being
> without loss/duplication of messages?
>
>
>
> On Thu, Jan 22, 2015 at 6:39 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Madhukar,
> >
> > We are currently working on improving MM to avoid data loss / duplicates,
> > details can be found in KAFKA-1650 / 1839 / 1840.
> >
> > For your concern, by saying "MM killed" do you mean it is shutdown or
> > hard-failed / crashed that can cause duplicates?
> >
> > Guozhang
> >
> > On Mon, Jan 19, 2015 at 10:51 PM, Madhukar Bharti <
> > bhartimadhukar@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I want to setup inter DC replication b/w Kafka clusters. Is there any
> > > inbuilt tool to do this?
> > >
> > > I already have tried MirrorMaker tool but the problem is, if MM killed
> > then
> > > some messages get duplicated. I don't want to duplicate the messages.
> > >
> > > Please suggest a way to do this. Please share your experience how you
> > have
> > > setup this.
> > >
> > >
> > > Thanks in Advance,
> > > Madhukar
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Thanks and Regards,
> Madhukar Bharti
>



-- 
-- Guozhang

Re: How to setup inter DC replication in Kafka 0.8.1.1

Posted by Madhukar Bharti <bh...@gmail.com>.

Thanks Guozhang for your reply.

Checked the details as mentioned above. Issue KAFKA-1650 has been fixed but
it will come with 0.8.3. :-(

By "MM Killed", it is hard-failed(kill -9 )/crashed due to system reboot or
any other reason. In that case some of the messages get duplicated.

Can you please suggest a way to do inter DC replication for time being
without loss/duplication of messages?

On Thu, Jan 22, 2015 at 6:39 AM, Guozhang Wang <wa...@gmail.com> wrote:

> Hi Madhukar,
>
> We are currently working on improving MM to avoid data loss / duplicates,
> details can be found in KAFKA-1650 / 1839 / 1840.
>
> For your concern, by saying "MM killed" do you mean it is shutdown or
> hard-failed / crashed that can cause duplicates?
>
> Guozhang
>
> On Mon, Jan 19, 2015 at 10:51 PM, Madhukar Bharti <
> bhartimadhukar@gmail.com>
> wrote:
>
> > Hi,
> >
> > I want to setup inter DC replication b/w Kafka clusters. Is there any
> > inbuilt tool to do this?
> >
> > I already have tried MirrorMaker tool but the problem is, if MM killed
> then
> > some messages get duplicated. I don't want to duplicate the messages.
> >
> > Please suggest a way to do this. Please share your experience how you
> have
> > setup this.
> >
> >
> > Thanks in Advance,
> > Madhukar
> >
>
>
>
> --
> -- Guozhang
>

-- 
Thanks and Regards,
Madhukar Bharti

Re: How to setup inter DC replication in Kafka 0.8.1.1

Posted by Guozhang Wang <wa...@gmail.com>.

Hi Madhukar,

We are currently working on improving MM to avoid data loss / duplicates,
details can be found in KAFKA-1650 / 1839 / 1840.

For your concern, by saying "MM killed" do you mean it is shutdown or
hard-failed / crashed that can cause duplicates?

Guozhang

On Mon, Jan 19, 2015 at 10:51 PM, Madhukar Bharti <bh...@gmail.com>
wrote:

> Hi,
>
> I want to setup inter DC replication b/w Kafka clusters. Is there any
> inbuilt tool to do this?
>
> I already have tried MirrorMaker tool but the problem is, if MM killed then
> some messages get duplicated. I don't want to duplicate the messages.
>
> Please suggest a way to do this. Please share your experience how you have
> setup this.
>
>
> Thanks in Advance,
> Madhukar
>

-- 
-- Guozhang