Kafka with Zookeeper behind AWS ELB

[Luigi Tagliamonte <lu...@gmail.com>]

Hello, Users!
I'm designing a Kafka deployment on AWS and it's my first time working with
Kafka and Zookeeper so I've collected a lot of info so far but also some
questions that I would love to submit to a much expert audience like you.

I have been experimenting with exhibitor and zookeeper in auto scaling
group and the exhibitor orchestration seems to work so far.

I was trying to find a way to configure zookeeper servers in Kafka conf and
do not have to reconfigure them in case a zookeeper node needs to be
replaced/dies, so i taught of course of using DNS but then I read that
zkclient library used by Kafka has this bug:
https://issues.apache.org/jira/browse/ZOOKEEPER-2184.

So I'm now thinking about using an ELB in front of the zookeeper cluster.
Teorically on how zookeeper client should work there should be no problem
but I'm wondering if any of you used that and how is the outcome?

Re: Kafka with Zookeeper behind AWS ELB

[Luigi Tagliamonte <lu...@gmail.com>]

Hello Pradeep,
thank you for sharing your experience, will certainly consider it.

On Thu, Jul 20, 2017 at 9:29 AM, Pradeep Gollakota <pr...@gmail.com>
wrote:

> Luigi,
>
> I strongly urge you to consider a 5 node ZK deployment. I've always done
> that in the past for resiliency during maintenance. In a 3 node cluster,
> you can only tolerate one "failure", so if you bring one node down for
> maintenance and another node crashes during said maintenance, your ZK
> cluster is down. All the deployments I've had were 5 nodes of ZK and 5
> nodes of Kafka.
>
> - Pradeep
>
> On Thu, Jul 20, 2017 at 9:12 AM, Luigi Tagliamonte <
> luigi.tagliamonte86@gmail.com> wrote:
>
> > Yes Andrey,
> > you can use an ENI without EIP on AWS if you only want a private address.
> >
> > After some consideration, I think that growing the zookeeper cluster more
> > than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3
> > servers in autoscaling and I will configure Kafka in using this 3 IPs.
> > In this way I can get rid of the additional ELB/Haproxy layer, if I will
> > ever need to grow the zk ensemble I will re-engineering the solution.
> >
> > I'm wondering if reusing an old IP on a brand new zk node will create
> > issues in the ensemble.
> > Is anybody here aware of possible drawbacks?
> >
> > On Wed, Jul 19, 2017 at 11:58 PM, Andrey Dyachkov <
> > andrey.dyachkov@gmail.com
> > > wrote:
> >
> > > The problem with EIP it is a public ip.
> > > Another option is to have the secondary interface attached to the
> > instance
> > > on start(or a bit later) with the private static ip, but we are
> > > investigating the possibility.
> > > On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte <
> > > luigi.tagliamonte86@gmail.com> wrote:
> > >
> > > > Hello Andrey,
> > > > I see that the ELB is not going to help directly with the bug, but
> > > > introduces a nice layer that makes zookeeper DNS management easier.
> > > > Introducing and ELB I don't have to deal with keep DNS in sync for
> all
> > > the
> > > > servers in the zk ensemble.
> > > > For the moment I can use an HAproxy with EIP and when the bug is
> > solved I
> > > > can move to ELB.
> > > > What do you think about it?
> > > > Regards
> > > > L.
> > > >
> > > > On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov <
> > > > andrey.dyachkov@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I have just posted almost the same question in dev list.
> > > > > Zookeeper client resolves address only once, on start, introducing
> > ELB
> > > > > won't really help here (ELBs can be replaced, which involved ip
> > > change),
> > > > > but I am eager to know if there is a solution for that.
> > > > >
> > > > > On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
> > > > > luigi.tagliamonte86@gmail.com> wrote:
> > > > >
> > > > > > Hello, Users!
> > > > > > I'm designing a Kafka deployment on AWS and it's my first time
> > > working
> > > > > with
> > > > > > Kafka and Zookeeper so I've collected a lot of info so far but
> also
> > > > some
> > > > > > questions that I would love to submit to a much expert audience
> > like
> > > > you.
> > > > > >
> > > > > > I have been experimenting with exhibitor and zookeeper in auto
> > > scaling
> > > > > > group and the exhibitor orchestration seems to work so far.
> > > > > >
> > > > > > I was trying to find a way to configure zookeeper servers in
> Kafka
> > > conf
> > > > > and
> > > > > > do not have to reconfigure them in case a zookeeper node needs to
> > be
> > > > > > replaced/dies, so i taught of course of using DNS but then I read
> > > that
> > > > > > zkclient library used by Kafka has this bug:
> > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
> > > > > >
> > > > > > So I'm now thinking about using an ELB in front of the zookeeper
> > > > cluster.
> > > > > > Teorically on how zookeeper client should work there should be no
> > > > problem
> > > > > > but I'm wondering if any of you used that and how is the outcome?
> > > > > >
> > > > > --
> > > > >
> > > > > With great enthusiasm,
> > > > > Andrey
> > > > >
> > > >
> > > --
> > >
> > > With great enthusiasm,
> > > Andrey
> > >
> >
>

Re: Kafka with Zookeeper behind AWS ELB

[Pradeep Gollakota <pr...@gmail.com>]

Luigi,

I strongly urge you to consider a 5 node ZK deployment. I've always done
that in the past for resiliency during maintenance. In a 3 node cluster,
you can only tolerate one "failure", so if you bring one node down for
maintenance and another node crashes during said maintenance, your ZK
cluster is down. All the deployments I've had were 5 nodes of ZK and 5
nodes of Kafka.

- Pradeep

On Thu, Jul 20, 2017 at 9:12 AM, Luigi Tagliamonte <
luigi.tagliamonte86@gmail.com> wrote:

> Yes Andrey,
> you can use an ENI without EIP on AWS if you only want a private address.
>
> After some consideration, I think that growing the zookeeper cluster more
> than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3
> servers in autoscaling and I will configure Kafka in using this 3 IPs.
> In this way I can get rid of the additional ELB/Haproxy layer, if I will
> ever need to grow the zk ensemble I will re-engineering the solution.
>
> I'm wondering if reusing an old IP on a brand new zk node will create
> issues in the ensemble.
> Is anybody here aware of possible drawbacks?
>
> On Wed, Jul 19, 2017 at 11:58 PM, Andrey Dyachkov <
> andrey.dyachkov@gmail.com
> > wrote:
>
> > The problem with EIP it is a public ip.
> > Another option is to have the secondary interface attached to the
> instance
> > on start(or a bit later) with the private static ip, but we are
> > investigating the possibility.
> > On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte <
> > luigi.tagliamonte86@gmail.com> wrote:
> >
> > > Hello Andrey,
> > > I see that the ELB is not going to help directly with the bug, but
> > > introduces a nice layer that makes zookeeper DNS management easier.
> > > Introducing and ELB I don't have to deal with keep DNS in sync for all
> > the
> > > servers in the zk ensemble.
> > > For the moment I can use an HAproxy with EIP and when the bug is
> solved I
> > > can move to ELB.
> > > What do you think about it?
> > > Regards
> > > L.
> > >
> > > On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov <
> > > andrey.dyachkov@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I have just posted almost the same question in dev list.
> > > > Zookeeper client resolves address only once, on start, introducing
> ELB
> > > > won't really help here (ELBs can be replaced, which involved ip
> > change),
> > > > but I am eager to know if there is a solution for that.
> > > >
> > > > On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
> > > > luigi.tagliamonte86@gmail.com> wrote:
> > > >
> > > > > Hello, Users!
> > > > > I'm designing a Kafka deployment on AWS and it's my first time
> > working
> > > > with
> > > > > Kafka and Zookeeper so I've collected a lot of info so far but also
> > > some
> > > > > questions that I would love to submit to a much expert audience
> like
> > > you.
> > > > >
> > > > > I have been experimenting with exhibitor and zookeeper in auto
> > scaling
> > > > > group and the exhibitor orchestration seems to work so far.
> > > > >
> > > > > I was trying to find a way to configure zookeeper servers in Kafka
> > conf
> > > > and
> > > > > do not have to reconfigure them in case a zookeeper node needs to
> be
> > > > > replaced/dies, so i taught of course of using DNS but then I read
> > that
> > > > > zkclient library used by Kafka has this bug:
> > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
> > > > >
> > > > > So I'm now thinking about using an ELB in front of the zookeeper
> > > cluster.
> > > > > Teorically on how zookeeper client should work there should be no
> > > problem
> > > > > but I'm wondering if any of you used that and how is the outcome?
> > > > >
> > > > --
> > > >
> > > > With great enthusiasm,
> > > > Andrey
> > > >
> > >
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>

Re: Kafka with Zookeeper behind AWS ELB

[Luigi Tagliamonte <lu...@gmail.com>]

Yes Andrey,
you can use an ENI without EIP on AWS if you only want a private address.

After some consideration, I think that growing the zookeeper cluster more
than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3
servers in autoscaling and I will configure Kafka in using this 3 IPs.
In this way I can get rid of the additional ELB/Haproxy layer, if I will
ever need to grow the zk ensemble I will re-engineering the solution.

I'm wondering if reusing an old IP on a brand new zk node will create
issues in the ensemble.
Is anybody here aware of possible drawbacks?

On Wed, Jul 19, 2017 at 11:58 PM, Andrey Dyachkov <andrey.dyachkov@gmail.com
> wrote:

> The problem with EIP it is a public ip.
> Another option is to have the secondary interface attached to the instance
> on start(or a bit later) with the private static ip, but we are
> investigating the possibility.
> On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte <
> luigi.tagliamonte86@gmail.com> wrote:
>
> > Hello Andrey,
> > I see that the ELB is not going to help directly with the bug, but
> > introduces a nice layer that makes zookeeper DNS management easier.
> > Introducing and ELB I don't have to deal with keep DNS in sync for all
> the
> > servers in the zk ensemble.
> > For the moment I can use an HAproxy with EIP and when the bug is solved I
> > can move to ELB.
> > What do you think about it?
> > Regards
> > L.
> >
> > On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov <
> > andrey.dyachkov@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I have just posted almost the same question in dev list.
> > > Zookeeper client resolves address only once, on start, introducing ELB
> > > won't really help here (ELBs can be replaced, which involved ip
> change),
> > > but I am eager to know if there is a solution for that.
> > >
> > > On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
> > > luigi.tagliamonte86@gmail.com> wrote:
> > >
> > > > Hello, Users!
> > > > I'm designing a Kafka deployment on AWS and it's my first time
> working
> > > with
> > > > Kafka and Zookeeper so I've collected a lot of info so far but also
> > some
> > > > questions that I would love to submit to a much expert audience like
> > you.
> > > >
> > > > I have been experimenting with exhibitor and zookeeper in auto
> scaling
> > > > group and the exhibitor orchestration seems to work so far.
> > > >
> > > > I was trying to find a way to configure zookeeper servers in Kafka
> conf
> > > and
> > > > do not have to reconfigure them in case a zookeeper node needs to be
> > > > replaced/dies, so i taught of course of using DNS but then I read
> that
> > > > zkclient library used by Kafka has this bug:
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
> > > >
> > > > So I'm now thinking about using an ELB in front of the zookeeper
> > cluster.
> > > > Teorically on how zookeeper client should work there should be no
> > problem
> > > > but I'm wondering if any of you used that and how is the outcome?
> > > >
> > > --
> > >
> > > With great enthusiasm,
> > > Andrey
> > >
> >
> --
>
> With great enthusiasm,
> Andrey
>

Re: Kafka with Zookeeper behind AWS ELB

[Andrey Dyachkov <an...@gmail.com>]

The problem with EIP it is a public ip.
Another option is to have the secondary interface attached to the instance
on start(or a bit later) with the private static ip, but we are
investigating the possibility.
On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte <
luigi.tagliamonte86@gmail.com> wrote:

> Hello Andrey,
> I see that the ELB is not going to help directly with the bug, but
> introduces a nice layer that makes zookeeper DNS management easier.
> Introducing and ELB I don't have to deal with keep DNS in sync for all the
> servers in the zk ensemble.
> For the moment I can use an HAproxy with EIP and when the bug is solved I
> can move to ELB.
> What do you think about it?
> Regards
> L.
>
> On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov <
> andrey.dyachkov@gmail.com>
> wrote:
>
> > Hi,
> > I have just posted almost the same question in dev list.
> > Zookeeper client resolves address only once, on start, introducing ELB
> > won't really help here (ELBs can be replaced, which involved ip change),
> > but I am eager to know if there is a solution for that.
> >
> > On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
> > luigi.tagliamonte86@gmail.com> wrote:
> >
> > > Hello, Users!
> > > I'm designing a Kafka deployment on AWS and it's my first time working
> > with
> > > Kafka and Zookeeper so I've collected a lot of info so far but also
> some
> > > questions that I would love to submit to a much expert audience like
> you.
> > >
> > > I have been experimenting with exhibitor and zookeeper in auto scaling
> > > group and the exhibitor orchestration seems to work so far.
> > >
> > > I was trying to find a way to configure zookeeper servers in Kafka conf
> > and
> > > do not have to reconfigure them in case a zookeeper node needs to be
> > > replaced/dies, so i taught of course of using DNS but then I read that
> > > zkclient library used by Kafka has this bug:
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
> > >
> > > So I'm now thinking about using an ELB in front of the zookeeper
> cluster.
> > > Teorically on how zookeeper client should work there should be no
> problem
> > > but I'm wondering if any of you used that and how is the outcome?
> > >
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>
-- 

With great enthusiasm,
Andrey

Re: Kafka with Zookeeper behind AWS ELB

[Luigi Tagliamonte <lu...@gmail.com>]

Hello Andrey,
I see that the ELB is not going to help directly with the bug, but
introduces a nice layer that makes zookeeper DNS management easier.
Introducing and ELB I don't have to deal with keep DNS in sync for all the
servers in the zk ensemble.
For the moment I can use an HAproxy with EIP and when the bug is solved I
can move to ELB.
What do you think about it?
Regards
L.

On Wed, Jul 19, 2017 at 2:16 PM, Andrey Dyachkov <an...@gmail.com>
wrote:

> Hi,
> I have just posted almost the same question in dev list.
> Zookeeper client resolves address only once, on start, introducing ELB
> won't really help here (ELBs can be replaced, which involved ip change),
> but I am eager to know if there is a solution for that.
>
> On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
> luigi.tagliamonte86@gmail.com> wrote:
>
> > Hello, Users!
> > I'm designing a Kafka deployment on AWS and it's my first time working
> with
> > Kafka and Zookeeper so I've collected a lot of info so far but also some
> > questions that I would love to submit to a much expert audience like you.
> >
> > I have been experimenting with exhibitor and zookeeper in auto scaling
> > group and the exhibitor orchestration seems to work so far.
> >
> > I was trying to find a way to configure zookeeper servers in Kafka conf
> and
> > do not have to reconfigure them in case a zookeeper node needs to be
> > replaced/dies, so i taught of course of using DNS but then I read that
> > zkclient library used by Kafka has this bug:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
> >
> > So I'm now thinking about using an ELB in front of the zookeeper cluster.
> > Teorically on how zookeeper client should work there should be no problem
> > but I'm wondering if any of you used that and how is the outcome?
> >
> --
>
> With great enthusiasm,
> Andrey
>

Re: Kafka with Zookeeper behind AWS ELB

[Andrey Dyachkov <an...@gmail.com>]

Hi,
I have just posted almost the same question in dev list.
Zookeeper client resolves address only once, on start, introducing ELB
won't really help here (ELBs can be replaced, which involved ip change),
but I am eager to know if there is a solution for that.

On Wed, 19 Jul 2017 at 23:08 Luigi Tagliamonte <
luigi.tagliamonte86@gmail.com> wrote:

> Hello, Users!
> I'm designing a Kafka deployment on AWS and it's my first time working with
> Kafka and Zookeeper so I've collected a lot of info so far but also some
> questions that I would love to submit to a much expert audience like you.
>
> I have been experimenting with exhibitor and zookeeper in auto scaling
> group and the exhibitor orchestration seems to work so far.
>
> I was trying to find a way to configure zookeeper servers in Kafka conf and
> do not have to reconfigure them in case a zookeeper node needs to be
> replaced/dies, so i taught of course of using DNS but then I read that
> zkclient library used by Kafka has this bug:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2184.
>
> So I'm now thinking about using an ELB in front of the zookeeper cluster.
> Teorically on how zookeeper client should work there should be no problem
> but I'm wondering if any of you used that and how is the outcome?
>
-- 

With great enthusiasm,
Andrey