You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sybrandy, Casey" <Ca...@Six3Systems.com> on 2014/10/16 04:16:31 UTC
Kafka/Zookeeper deployment Questions
Hello,
We're looking into deploying Kafka and Zookeeper into an environment where we want things to be as easy to stand up and administer. To do this, we're looking into using Consul, or similar, and Confd to try to make this as automatic as possible. I was wondering if anyone had an experience in this area. My major concern is reconfiguring Kafka as, in my experience, is making sure we don't end up losing messages.
Also, can kafka and zookeeper be reconfigured in a rolling manner? In other words, if I change the number of partitions, can I restart the brokers one at a time so that I can continue processing data?
Thanks.
Re: Kafka/Zookeeper deployment Questions
Posted by Roger Hoover <ro...@gmail.com>.
Thank you, Casey. Both of those features sound pretty useful.
On Fri, Oct 17, 2014 at 10:16 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Roger,
>
> My understanding of both, beyond what Zookeeper already does, are:
>
> 1. Consul can be used to monitor a service and report it's status. This
> can be very useful for knowing if a service, such as Zookeeper of Kafka,
> goes down. This can be done through a built-in web interface.
> 2. Confd leverages Consul, or etcd, to propogate changes to a service and
> restart it if necessary. So, if we change a broker specific setting, we
> can put the change in Consul and have Confd automatically modify the config
> files on the broker nodes and restart the service as needed.
>
> My knowledge in this area is a bit limited as I haven't used either. I'm
> working with someone who is and wanted to ask people about this so that we
> can learn what works and what doesn't.
>
> _______________________________________
> From: Roger Hoover [roger.hoover@gmail.com]
> Sent: Friday, October 17, 2014 12:26 PM
> To: users@kafka.apache.org
> Subject: Re: Kafka/Zookeeper deployment Questions
>
> Casey,
>
> Could you describe a little more about how these would help manage a
> cluster?
>
> My understanding is that Consul provides service discovery and leader
> election. Kafka already uses ZooKeeper for brokers to discover each other
> and elect partition leaders. Kafka high-level consumers use ZK to divide
> up topic partitions amongst themselves.
>
> I'm not able to see how Consul +/or confd will help.
>
> Cheers,
>
> Roger
>
RE: Kafka/Zookeeper deployment Questions
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Roger,
My understanding of both, beyond what Zookeeper already does, are:
1. Consul can be used to monitor a service and report it's status. This can be very useful for knowing if a service, such as Zookeeper of Kafka, goes down. This can be done through a built-in web interface.
2. Confd leverages Consul, or etcd, to propogate changes to a service and restart it if necessary. So, if we change a broker specific setting, we can put the change in Consul and have Confd automatically modify the config files on the broker nodes and restart the service as needed.
My knowledge in this area is a bit limited as I haven't used either. I'm working with someone who is and wanted to ask people about this so that we can learn what works and what doesn't.
_______________________________________
From: Roger Hoover [roger.hoover@gmail.com]
Sent: Friday, October 17, 2014 12:26 PM
To: users@kafka.apache.org
Subject: Re: Kafka/Zookeeper deployment Questions
Casey,
Could you describe a little more about how these would help manage a
cluster?
My understanding is that Consul provides service discovery and leader
election. Kafka already uses ZooKeeper for brokers to discover each other
and elect partition leaders. Kafka high-level consumers use ZK to divide
up topic partitions amongst themselves.
I'm not able to see how Consul +/or confd will help.
Cheers,
Roger
Re: Kafka/Zookeeper deployment Questions
Posted by Roger Hoover <ro...@gmail.com>.
Casey,
Could you describe a little more about how these would help manage a
cluster?
My understanding is that Consul provides service discovery and leader
election. Kafka already uses ZooKeeper for brokers to discover each other
and elect partition leaders. Kafka high-level consumers use ZK to divide
up topic partitions amongst themselves.
I'm not able to see how Consul +/or confd will help.
Cheers,
Roger
On Fri, Oct 17, 2014 at 5:16 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Neha,
>
> Thanks. I'd still love to know if anyone has used Consul and/or Confd to
> manage a cluster.
>
> Casey
>
> ________________________________________
> From: Neha Narkhede [neha.narkhede@gmail.com]
> Sent: Thursday, October 16, 2014 9:54 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka/Zookeeper deployment Questions
>
> In other words, if I change the number of partitions, can I restart the
> brokers one at a time so that I can continue processing data?
>
> Changing the # of partitions is an online operation and doesn't require
> restarting the brokers. However, any other configuration (with the
> exception of a few operations) that requires a broker restart can be done
> in a rolling manner.
>
> On Wed, Oct 15, 2014 at 7:16 PM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > We're looking into deploying Kafka and Zookeeper into an environment
> where
> > we want things to be as easy to stand up and administer. To do this,
> we're
> > looking into using Consul, or similar, and Confd to try to make this as
> > automatic as possible. I was wondering if anyone had an experience in
> this
> > area. My major concern is reconfiguring Kafka as, in my experience, is
> > making sure we don't end up losing messages.
> >
> > Also, can kafka and zookeeper be reconfigured in a rolling manner? In
> > other words, if I change the number of partitions, can I restart the
> > brokers one at a time so that I can continue processing data?
> >
> > Thanks.
>
RE: Kafka/Zookeeper deployment Questions
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Neha,
Thanks. I'd still love to know if anyone has used Consul and/or Confd to manage a cluster.
Casey
________________________________________
From: Neha Narkhede [neha.narkhede@gmail.com]
Sent: Thursday, October 16, 2014 9:54 AM
To: users@kafka.apache.org
Subject: Re: Kafka/Zookeeper deployment Questions
In other words, if I change the number of partitions, can I restart the
brokers one at a time so that I can continue processing data?
Changing the # of partitions is an online operation and doesn't require
restarting the brokers. However, any other configuration (with the
exception of a few operations) that requires a broker restart can be done
in a rolling manner.
On Wed, Oct 15, 2014 at 7:16 PM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> We're looking into deploying Kafka and Zookeeper into an environment where
> we want things to be as easy to stand up and administer. To do this, we're
> looking into using Consul, or similar, and Confd to try to make this as
> automatic as possible. I was wondering if anyone had an experience in this
> area. My major concern is reconfiguring Kafka as, in my experience, is
> making sure we don't end up losing messages.
>
> Also, can kafka and zookeeper be reconfigured in a rolling manner? In
> other words, if I change the number of partitions, can I restart the
> brokers one at a time so that I can continue processing data?
>
> Thanks.
Re: Kafka/Zookeeper deployment Questions
Posted by Neha Narkhede <ne...@gmail.com>.
In other words, if I change the number of partitions, can I restart the
brokers one at a time so that I can continue processing data?
Changing the # of partitions is an online operation and doesn't require
restarting the brokers. However, any other configuration (with the
exception of a few operations) that requires a broker restart can be done
in a rolling manner.
On Wed, Oct 15, 2014 at 7:16 PM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> We're looking into deploying Kafka and Zookeeper into an environment where
> we want things to be as easy to stand up and administer. To do this, we're
> looking into using Consul, or similar, and Confd to try to make this as
> automatic as possible. I was wondering if anyone had an experience in this
> area. My major concern is reconfiguring Kafka as, in my experience, is
> making sure we don't end up losing messages.
>
> Also, can kafka and zookeeper be reconfigured in a rolling manner? In
> other words, if I change the number of partitions, can I restart the
> brokers one at a time so that I can continue processing data?
>
> Thanks.