You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Xuyen On <xo...@ancestry.com> on 2013/09/10 02:39:59 UTC
Failover for Zookeeper and Kafka
Hi everyone,
I am trying to setup a Kafka cluster and have a couple of questions about failover.
Has anyone deployed more than one zookeeper for a single Kafka cluster and have high availability so if one zookeeper node goes down, the cluster automatically fails over to a backup zookeeper node? If so, how is this done?
My second question is how can I set up for automatic failover if I have a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what do I need to do in order for my producers and consumers to automatically fail over to the backup mirror Kafka cluster. Do I need to code this into my producers and consumers, should I setup a DNS redirect in case the main Kafka cluster goes down to point to the mirror cluster, or is there some other configuration that I can do?
Thanks,
Xuyen
Re: Failover for Zookeeper and Kafka
Posted by Jun Rao <ju...@gmail.com>.
Yes, we have been using MirrorMaker at LinkedIn. You may need to tweak the
socket buffer size to amortize the cross DC network latency.
Thanks,
Jun
On Mon, Nov 25, 2013 at 1:34 AM, Arjun <ar...@socialtwist.com> wrote:
> Hi ,
>
> I just saw your post. Can i know weather you are successful in doing this?
> i am planning to do the same for our environment. do i need to take care of
> anything before i start the testing. If you can let me know any thing which
> would make my life easier i would be pretty much grateful.
>
> Thanks
> Arjun Narasimha Kota
>
Failover for Zookeeper and Kafka
Posted by Arjun <ar...@socialtwist.com>.
Hi ,
I just saw your post. Can i know weather you are successful in doing
this? i am planning to do the same for our environment. do i need to
take care of anything before i start the testing. If you can let me know
any thing which would make my life easier i would be pretty much grateful.
Thanks
Arjun Narasimha Kota
Re: Failover for Zookeeper and Kafka
Posted by Jun Rao <ju...@gmail.com>.
For asynchronous mirroring, you can take a look at
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29
Thanks,
Jun
On Mon, Sep 9, 2013 at 11:30 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi Joe,
>
> Yes I think we'd need to setup 3 zookeeper nodes in order to support a
> single node zookeeper failure. Was just wondering what other people have
> done in their production system. Is it worth it to setup 3 zookeeper nodes
> for a 3 - 5 node kafka cluster or would a single zookeeper node be enough?
>
> On my second question we are using Kafka 0.7 right so we do not have the
> replication feature yet. I have read about the mirroring feature you
> mentioned but I am hoping someone can help me find a way to setup an
> active-active 0.7 Kafka set of clusters (one main, one hot backup) so that
> when the main cluster goes down, the producers and consumers will
> automatically switch over to the backup Kafka cluster. I've read somewhere
> that LinkedIn has an active-active setup and was wondering how they are
> able to do this if possible with 0.7 Kafka.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joe Stein [mailto:cryptcom@gmail.com]
> Sent: Monday, September 09, 2013 5:56 PM
> To: users@kafka.apache.org
> Subject: Re: Failover for Zookeeper and Kafka
>
> You want to setup a Zookeeper ensemble ( always an odd number of servers,
> three is often acceptable )
> http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
>
> And use Kafka 0.8 replication
> http://kafka.apache.org/documentation.html#replication in addition if you
> want to mirror the cluster like in another data center you could also use
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
> ********************************************/
>
>
> On Sep 9, 2013, at 8:39 PM, Xuyen On <xo...@ancestry.com> wrote:
>
> > Hi everyone,
> >
> > I am trying to setup a Kafka cluster and have a couple of questions
> about failover.
> >
> > Has anyone deployed more than one zookeeper for a single Kafka cluster
> and have high availability so if one zookeeper node goes down, the cluster
> automatically fails over to a backup zookeeper node? If so, how is this
> done?
> >
> > My second question is how can I set up for automatic failover if I have
> a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down,
> what do I need to do in order for my producers and consumers to
> automatically fail over to the backup mirror Kafka cluster. Do I need to
> code this into my producers and consumers, should I setup a DNS redirect in
> case the main Kafka cluster goes down to point to the mirror cluster, or is
> there some other configuration that I can do?
> >
> > Thanks,
> >
> > Xuyen
> >
> >
>
>
RE: Failover for Zookeeper and Kafka
Posted by Xuyen On <xo...@ancestry.com>.
Hi Joe,
Yes I think we'd need to setup 3 zookeeper nodes in order to support a single node zookeeper failure. Was just wondering what other people have done in their production system. Is it worth it to setup 3 zookeeper nodes for a 3 - 5 node kafka cluster or would a single zookeeper node be enough?
On my second question we are using Kafka 0.7 right so we do not have the replication feature yet. I have read about the mirroring feature you mentioned but I am hoping someone can help me find a way to setup an active-active 0.7 Kafka set of clusters (one main, one hot backup) so that when the main cluster goes down, the producers and consumers will automatically switch over to the backup Kafka cluster. I've read somewhere that LinkedIn has an active-active setup and was wondering how they are able to do this if possible with 0.7 Kafka.
Thanks,
Xuyen
-----Original Message-----
From: Joe Stein [mailto:cryptcom@gmail.com]
Sent: Monday, September 09, 2013 5:56 PM
To: users@kafka.apache.org
Subject: Re: Failover for Zookeeper and Kafka
You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable ) http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Sep 9, 2013, at 8:39 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi everyone,
>
> I am trying to setup a Kafka cluster and have a couple of questions about failover.
>
> Has anyone deployed more than one zookeeper for a single Kafka cluster and have high availability so if one zookeeper node goes down, the cluster automatically fails over to a backup zookeeper node? If so, how is this done?
>
> My second question is how can I set up for automatic failover if I have a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what do I need to do in order for my producers and consumers to automatically fail over to the backup mirror Kafka cluster. Do I need to code this into my producers and consumers, should I setup a DNS redirect in case the main Kafka cluster goes down to point to the mirror cluster, or is there some other configuration that I can do?
>
> Thanks,
>
> Xuyen
>
>
Re: Failover for Zookeeper and Kafka
Posted by Joe Stein <cr...@gmail.com>.
You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable )
http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Sep 9, 2013, at 8:39 PM, Xuyen On <xo...@ancestry.com> wrote:
> Hi everyone,
>
> I am trying to setup a Kafka cluster and have a couple of questions about failover.
>
> Has anyone deployed more than one zookeeper for a single Kafka cluster and have high availability so if one zookeeper node goes down, the cluster automatically fails over to a backup zookeeper node? If so, how is this done?
>
> My second question is how can I set up for automatic failover if I have a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what do I need to do in order for my producers and consumers to automatically fail over to the backup mirror Kafka cluster. Do I need to code this into my producers and consumers, should I setup a DNS redirect in case the main Kafka cluster goes down to point to the mirror cluster, or is there some other configuration that I can do?
>
> Thanks,
>
> Xuyen
>
>