You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by naresh Goud <na...@gmail.com> on 2018/11/14 18:48:41 UTC

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Static IP. Buying static IP may help. I am not aws expert

On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com> wrote:

> Hello Kafka experts,
>
> We are running Kafka on AWS, main question is what is the best way to
> retain broker.id on new instance spun-up in-place of instance/broker
> failed.
>
> We are currently running Kafka in AWS with broker.id gets auto generated.
> But we are having issues when a broker is failed, new broker/instance
> spun-up in AWS get assigned with new broker.id. The issue is, with this
> approach, we need to re-assign the topics/replications on to the new broker
> manually.
>
> We learned that, replication can be auto resolved by Kafka, if we can
> manage to get the same broker.id on the new AWS instance spun-up in-place
> of failed broker/instance.
>
> I have read, we can set broker.id.generation.enable= false, but what is the
> best way to identify and retain the broker.id? Any links/help is
> appreciated.
> Thanks and Regards,
> Cnu
>
-- 
Thanks,
Naresh
www.linkedin.com/in/naresh-dulam
http://hadoopandspark.blogspot.com/

RE: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Tim Ward <ti...@origamienergy.com>.
Isn't that what stateful sets are for, giving you a predictable pod name from which you can derive the ID?

Tim Ward

-----Original Message-----
From: Srinivas Rapolu <cn...@gmail.com>
Sent: 15 November 2018 17:42
To: users@kafka.apache.org
Subject: Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Yes, I understand we need to specify the required broker.id in
server.properties/meta.properties file in-order to retain the broker.id
which was failed.

We can write custom script to query the zookeeper on launch process to get
the broker.id needs to be set.

Do we have any standards/script/way defined/preferred for doing this or
suggested by Kafka experts if we are not using EBS?

Thanks and Regards,
Srinivas

On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io> wrote:

> Srinivas,
>
> You might want to check out the AWS best practices from this blog post:
>
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
>
> Kafka broker ids by default are auto-generated, but you can also specify
> values (in server.properties file). By having static broker ids, it's
> easier to recover in general. The blog post above mentions it in more
> details.
>
> Good luck.
>
> On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:
>
> > If at all you were hell bent on doing this, you could use zookeeper to
> find
> > out the health of current brokers along with their broker id. That should
> > help re-spin/start the unhealthy instance with same instance id.
> >
> > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com>
> > wrote:
> >
> > > having all this stored in DB is getting too complicated, especially
> with
> > > instance level storage and not EBS.
> > >
> > > I am sure there should be easy way to retain the old broker.id for new
> > AWS
> > > instance spun-up for auto replication.
> > >
> > > Any other ideas/help is appreciated.
> > >
> > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <en...@gmail.com>
> > > wrote:
> > >
> > > > The general answer depends on what control plane software is taking
> > care
> > > of
> > > > your Kafka deployment. You probably have a layer that launches Kafka
> > > > instances and monitors their health, right? If so, that layer should
> > take
> > > > care of the mapping between instances and broker IDs and keep that
> in a
> > > > table persisted somewhere (e.g., DynamoDB).
> > > >
> > > > Eno
> > > >
> > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com>
> > > > wrote:
> > > >
> > > > > EBS is one of the option. But we use instance level storage where
> we
> > > > loose
> > > > > all data as soon as we have a broker failed in AWS.
> > > > >
> > > > > In such scenario, anyone has better launch script or cofiguration
> can
> > > be
> > > > > executed on new broker to retain the old id not conflicting with
> > > existing
> > > > > broker ids.
> > > > >
> > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > andrey.dyachkov@gmail.com
> > > > > wrote:
> > > > >
> > > > > > You can attach EBS volume, which will store data and
> metadata(e.g.
> > > > broker
> > > > > > id), and then attach it to the new AWS instance and start Kafka,
> it
> > > > will
> > > > > > pick the broker id plus you won’t need to rebalance the cluster.
> > > > > >
> > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > nareshgoud.dulam@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > > >
> > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > cnu.text@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka experts,
> > > > > > > >
> > > > > > > > We are running Kafka on AWS, main question is what is the
> best
> > > way
> > > > to
> > > > > > > > retain broker.id on new instance spun-up in-place of
> > > > instance/broker
> > > > > > > > failed.
> > > > > > > >
> > > > > > > > We are currently running Kafka in AWS with broker.id gets
> auto
> > > > > > > generated.
> > > > > > > > But we are having issues when a broker is failed, new
> > > > broker/instance
> > > > > > > > spun-up in AWS get assigned with new broker.id. The issue
> is,
> > > with
> > > > > > this
> > > > > > > > approach, we need to re-assign the topics/replications on to
> > the
> > > > new
> > > > > > > broker
> > > > > > > > manually.
> > > > > > > >
> > > > > > > > We learned that, replication can be auto resolved by Kafka,
> if
> > we
> > > > can
> > > > > > > > manage to get the same broker.id on the new AWS instance
> > spun-up
> > > > > > > in-place
> > > > > > > > of failed broker/instance.
> > > > > > > >
> > > > > > > > I have read, we can set broker.id.generation.enable= false,
> but
> > > > what
> > > > > is
> > > > > > > the
> > > > > > > > best way to identify and retain the broker.id? Any
> links/help
> > is
> > > > > > > > appreciated.
> > > > > > > > Thanks and Regards,
> > > > > > > > Cnu
> > > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Naresh
> > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Andrey
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Thanks and Regards,
> > Amit Pal
> >
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>
The contents of this email and any attachment are confidential to the intended recipient(s). If you are not an intended recipient: (i) do not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact the sender immediately; and (iii) delete this email. Our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy Limited (company number 8619644); Origami Storage Limited (company number 10436515) and OSSPV001 Limited (company number 10933403), each registered in England and each with a registered office at: Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
 By looking at the responses: It looks like there is *NO *standard
solution, we will have to come up with some custom/manual script..... any
one has any other suggestions?

On Fri, Nov 16, 2018 at 11:18 AM Srinivas Rapolu <cn...@gmail.com> wrote:

> Yes, that is the whole point of discussion, replication factor is more
> than 1. So keeping it in instance storage, we will lose the one copy of
> data, but retaining broker-id will auto-replicate the topic/partitions to
> new instance as zookeeper has brokerid-topic-partition assignment mapping.
>
> By looking at the responses: It looks like there is standard solution, we
> will have to come up with some manual script..... any one has any other
> suggestions?
>
> On Fri, Nov 16, 2018 at 10:08 AM Kaufman Ng <ka...@confluent.io> wrote:
>
>> Yes data will be lost when you use instance storage. But keeping the
>> *same*
>> broker id will allow the new broker instance to easily replicate data from
>> other brokers, therefore restoring back to the previous fully replicated
>> state.
>>
>> Srinivas, do your topics have replicator factor > 1? If yes, data can be
>> recovered from other brokers.
>>
>> On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ry...@gmail.com>
>> wrote:
>>
>> > > if we are not using EBS
>> >
>> > In that case, what's the point of keeping the broker id? The data will
>> be
>> > lost anyway right?
>> >
>> >
>> > On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cn...@gmail.com>
>> wrote:
>> >
>> > > Yes, I understand we need to specify the required broker.id in
>> > > server.properties/meta.properties file in-order to retain the
>> broker.id
>> > > which was failed.
>> > >
>> > > We can write custom script to query the zookeeper on launch process to
>> > get
>> > > the broker.id needs to be set.
>> > >
>> > > Do we have any standards/script/way defined/preferred for doing this
>> or
>> > > suggested by Kafka experts if we are not using EBS?
>> > >
>> > > Thanks and Regards,
>> > > Srinivas
>> > >
>> > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io>
>> wrote:
>> > >
>> > > > Srinivas,
>> > > >
>> > > > You might want to check out the AWS best practices from this blog
>> post:
>> > > >
>> > > >
>> > >
>> >
>> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
>> > > >
>> > > > Kafka broker ids by default are auto-generated, but you can also
>> > specify
>> > > > values (in server.properties file). By having static broker ids,
>> it's
>> > > > easier to recover in general. The blog post above mentions it in
>> more
>> > > > details.
>> > > >
>> > > > Good luck.
>> > > >
>> > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com>
>> wrote:
>> > > >
>> > > > > If at all you were hell bent on doing this, you could use
>> zookeeper
>> > to
>> > > > find
>> > > > > out the health of current brokers along with their broker id. That
>> > > should
>> > > > > help re-spin/start the unhealthy instance with same instance id.
>> > > > >
>> > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <
>> cnu.text@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > having all this stored in DB is getting too complicated,
>> especially
>> > > > with
>> > > > > > instance level storage and not EBS.
>> > > > > >
>> > > > > > I am sure there should be easy way to retain the old broker.id
>> for
>> > > new
>> > > > > AWS
>> > > > > > instance spun-up for auto replication.
>> > > > > >
>> > > > > > Any other ideas/help is appreciated.
>> > > > > >
>> > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <
>> > eno.thereska@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > The general answer depends on what control plane software is
>> > taking
>> > > > > care
>> > > > > > of
>> > > > > > > your Kafka deployment. You probably have a layer that launches
>> > > Kafka
>> > > > > > > instances and monitors their health, right? If so, that layer
>> > > should
>> > > > > take
>> > > > > > > care of the mapping between instances and broker IDs and keep
>> > that
>> > > > in a
>> > > > > > > table persisted somewhere (e.g., DynamoDB).
>> > > > > > >
>> > > > > > > Eno
>> > > > > > >
>> > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
>> > > cnu.text@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > EBS is one of the option. But we use instance level storage
>> > where
>> > > > we
>> > > > > > > loose
>> > > > > > > > all data as soon as we have a broker failed in AWS.
>> > > > > > > >
>> > > > > > > > In such scenario, anyone has better launch script or
>> > cofiguration
>> > > > can
>> > > > > > be
>> > > > > > > > executed on new broker to retain the old id not conflicting
>> > with
>> > > > > > existing
>> > > > > > > > broker ids.
>> > > > > > > >
>> > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
>> > > > > > > andrey.dyachkov@gmail.com
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > You can attach EBS volume, which will store data and
>> > > > metadata(e.g.
>> > > > > > > broker
>> > > > > > > > > id), and then attach it to the new AWS instance and start
>> > > Kafka,
>> > > > it
>> > > > > > > will
>> > > > > > > > > pick the broker id plus you won’t need to rebalance the
>> > > cluster.
>> > > > > > > > >
>> > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
>> > > > > > nareshgoud.dulam@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Static IP. Buying static IP may help. I am not aws
>> expert
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
>> > > > > > cnu.text@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hello Kafka experts,
>> > > > > > > > > > >
>> > > > > > > > > > > We are running Kafka on AWS, main question is what is
>> the
>> > > > best
>> > > > > > way
>> > > > > > > to
>> > > > > > > > > > > retain broker.id on new instance spun-up in-place of
>> > > > > > > instance/broker
>> > > > > > > > > > > failed.
>> > > > > > > > > > >
>> > > > > > > > > > > We are currently running Kafka in AWS with broker.id
>> > gets
>> > > > auto
>> > > > > > > > > > generated.
>> > > > > > > > > > > But we are having issues when a broker is failed, new
>> > > > > > > broker/instance
>> > > > > > > > > > > spun-up in AWS get assigned with new broker.id. The
>> > issue
>> > > > is,
>> > > > > > with
>> > > > > > > > > this
>> > > > > > > > > > > approach, we need to re-assign the
>> topics/replications on
>> > > to
>> > > > > the
>> > > > > > > new
>> > > > > > > > > > broker
>> > > > > > > > > > > manually.
>> > > > > > > > > > >
>> > > > > > > > > > > We learned that, replication can be auto resolved by
>> > Kafka,
>> > > > if
>> > > > > we
>> > > > > > > can
>> > > > > > > > > > > manage to get the same broker.id on the new AWS
>> instance
>> > > > > spun-up
>> > > > > > > > > > in-place
>> > > > > > > > > > > of failed broker/instance.
>> > > > > > > > > > >
>> > > > > > > > > > > I have read, we can set broker.id.generation.enable=
>> > false,
>> > > > but
>> > > > > > > what
>> > > > > > > > is
>> > > > > > > > > > the
>> > > > > > > > > > > best way to identify and retain the broker.id? Any
>> > > > links/help
>> > > > > is
>> > > > > > > > > > > appreciated.
>> > > > > > > > > > > Thanks and Regards,
>> > > > > > > > > > > Cnu
>> > > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Naresh
>> > > > > > > > > > www.linkedin.com/in/naresh-dulam
>> > > > > > > > > > http://hadoopandspark.blogspot.com/
>> > > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Thanks,
>> > > > > > > > > Andrey
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Thanks and Regards,
>> > > > > Amit Pal
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Kaufman Ng
>> > > > +1 646 961 8063
>> > > > Solutions Architect | Confluent | www.confluent.io
>> > > >
>> > >
>> >
>>
>>
>> --
>> Kaufman Ng
>> +1 646 961 8063
>> Solutions Architect | Confluent | www.confluent.io
>>
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
Yes, that is the whole point of discussion, replication factor is more than
1. So keeping it in instance storage, we will lose the one copy of data,
but retaining broker-id will auto-replicate the topic/partitions to new
instance as zookeeper has brokerid-topic-partition assignment mapping.

By looking at the responses: It looks like there is standard solution, we
will have to come up with some manual script..... any one has any other
suggestions?

On Fri, Nov 16, 2018 at 10:08 AM Kaufman Ng <ka...@confluent.io> wrote:

> Yes data will be lost when you use instance storage. But keeping the *same*
> broker id will allow the new broker instance to easily replicate data from
> other brokers, therefore restoring back to the previous fully replicated
> state.
>
> Srinivas, do your topics have replicator factor > 1? If yes, data can be
> recovered from other brokers.
>
> On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ry...@gmail.com>
> wrote:
>
> > > if we are not using EBS
> >
> > In that case, what's the point of keeping the broker id? The data will be
> > lost anyway right?
> >
> >
> > On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cn...@gmail.com>
> wrote:
> >
> > > Yes, I understand we need to specify the required broker.id in
> > > server.properties/meta.properties file in-order to retain the
> broker.id
> > > which was failed.
> > >
> > > We can write custom script to query the zookeeper on launch process to
> > get
> > > the broker.id needs to be set.
> > >
> > > Do we have any standards/script/way defined/preferred for doing this or
> > > suggested by Kafka experts if we are not using EBS?
> > >
> > > Thanks and Regards,
> > > Srinivas
> > >
> > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io>
> wrote:
> > >
> > > > Srinivas,
> > > >
> > > > You might want to check out the AWS best practices from this blog
> post:
> > > >
> > > >
> > >
> >
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
> > > >
> > > > Kafka broker ids by default are auto-generated, but you can also
> > specify
> > > > values (in server.properties file). By having static broker ids, it's
> > > > easier to recover in general. The blog post above mentions it in more
> > > > details.
> > > >
> > > > Good luck.
> > > >
> > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:
> > > >
> > > > > If at all you were hell bent on doing this, you could use zookeeper
> > to
> > > > find
> > > > > out the health of current brokers along with their broker id. That
> > > should
> > > > > help re-spin/start the unhealthy instance with same instance id.
> > > > >
> > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <
> cnu.text@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > having all this stored in DB is getting too complicated,
> especially
> > > > with
> > > > > > instance level storage and not EBS.
> > > > > >
> > > > > > I am sure there should be easy way to retain the old broker.id
> for
> > > new
> > > > > AWS
> > > > > > instance spun-up for auto replication.
> > > > > >
> > > > > > Any other ideas/help is appreciated.
> > > > > >
> > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <
> > eno.thereska@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > The general answer depends on what control plane software is
> > taking
> > > > > care
> > > > > > of
> > > > > > > your Kafka deployment. You probably have a layer that launches
> > > Kafka
> > > > > > > instances and monitors their health, right? If so, that layer
> > > should
> > > > > take
> > > > > > > care of the mapping between instances and broker IDs and keep
> > that
> > > > in a
> > > > > > > table persisted somewhere (e.g., DynamoDB).
> > > > > > >
> > > > > > > Eno
> > > > > > >
> > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
> > > cnu.text@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > EBS is one of the option. But we use instance level storage
> > where
> > > > we
> > > > > > > loose
> > > > > > > > all data as soon as we have a broker failed in AWS.
> > > > > > > >
> > > > > > > > In such scenario, anyone has better launch script or
> > cofiguration
> > > > can
> > > > > > be
> > > > > > > > executed on new broker to retain the old id not conflicting
> > with
> > > > > > existing
> > > > > > > > broker ids.
> > > > > > > >
> > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > > > > andrey.dyachkov@gmail.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > You can attach EBS volume, which will store data and
> > > > metadata(e.g.
> > > > > > > broker
> > > > > > > > > id), and then attach it to the new AWS instance and start
> > > Kafka,
> > > > it
> > > > > > > will
> > > > > > > > > pick the broker id plus you won’t need to rebalance the
> > > cluster.
> > > > > > > > >
> > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > > > > nareshgoud.dulam@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > > > > cnu.text@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello Kafka experts,
> > > > > > > > > > >
> > > > > > > > > > > We are running Kafka on AWS, main question is what is
> the
> > > > best
> > > > > > way
> > > > > > > to
> > > > > > > > > > > retain broker.id on new instance spun-up in-place of
> > > > > > > instance/broker
> > > > > > > > > > > failed.
> > > > > > > > > > >
> > > > > > > > > > > We are currently running Kafka in AWS with broker.id
> > gets
> > > > auto
> > > > > > > > > > generated.
> > > > > > > > > > > But we are having issues when a broker is failed, new
> > > > > > > broker/instance
> > > > > > > > > > > spun-up in AWS get assigned with new broker.id. The
> > issue
> > > > is,
> > > > > > with
> > > > > > > > > this
> > > > > > > > > > > approach, we need to re-assign the topics/replications
> on
> > > to
> > > > > the
> > > > > > > new
> > > > > > > > > > broker
> > > > > > > > > > > manually.
> > > > > > > > > > >
> > > > > > > > > > > We learned that, replication can be auto resolved by
> > Kafka,
> > > > if
> > > > > we
> > > > > > > can
> > > > > > > > > > > manage to get the same broker.id on the new AWS
> instance
> > > > > spun-up
> > > > > > > > > > in-place
> > > > > > > > > > > of failed broker/instance.
> > > > > > > > > > >
> > > > > > > > > > > I have read, we can set broker.id.generation.enable=
> > false,
> > > > but
> > > > > > > what
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > best way to identify and retain the broker.id? Any
> > > > links/help
> > > > > is
> > > > > > > > > > > appreciated.
> > > > > > > > > > > Thanks and Regards,
> > > > > > > > > > > Cnu
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Thanks,
> > > > > > > > > > Naresh
> > > > > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks,
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > > Amit Pal
> > > > >
> > > >
> > > >
> > > > --
> > > > Kaufman Ng
> > > > +1 646 961 8063
> > > > Solutions Architect | Confluent | www.confluent.io
> > > >
> > >
> >
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Kaufman Ng <ka...@confluent.io>.
Yes data will be lost when you use instance storage. But keeping the *same*
broker id will allow the new broker instance to easily replicate data from
other brokers, therefore restoring back to the previous fully replicated
state.

Srinivas, do your topics have replicator factor > 1? If yes, data can be
recovered from other brokers.

On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ry...@gmail.com> wrote:

> > if we are not using EBS
>
> In that case, what's the point of keeping the broker id? The data will be
> lost anyway right?
>
>
> On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cn...@gmail.com> wrote:
>
> > Yes, I understand we need to specify the required broker.id in
> > server.properties/meta.properties file in-order to retain the broker.id
> > which was failed.
> >
> > We can write custom script to query the zookeeper on launch process to
> get
> > the broker.id needs to be set.
> >
> > Do we have any standards/script/way defined/preferred for doing this or
> > suggested by Kafka experts if we are not using EBS?
> >
> > Thanks and Regards,
> > Srinivas
> >
> > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io> wrote:
> >
> > > Srinivas,
> > >
> > > You might want to check out the AWS best practices from this blog post:
> > >
> > >
> >
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
> > >
> > > Kafka broker ids by default are auto-generated, but you can also
> specify
> > > values (in server.properties file). By having static broker ids, it's
> > > easier to recover in general. The blog post above mentions it in more
> > > details.
> > >
> > > Good luck.
> > >
> > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:
> > >
> > > > If at all you were hell bent on doing this, you could use zookeeper
> to
> > > find
> > > > out the health of current brokers along with their broker id. That
> > should
> > > > help re-spin/start the unhealthy instance with same instance id.
> > > >
> > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com>
> > > > wrote:
> > > >
> > > > > having all this stored in DB is getting too complicated, especially
> > > with
> > > > > instance level storage and not EBS.
> > > > >
> > > > > I am sure there should be easy way to retain the old broker.id for
> > new
> > > > AWS
> > > > > instance spun-up for auto replication.
> > > > >
> > > > > Any other ideas/help is appreciated.
> > > > >
> > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <
> eno.thereska@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > The general answer depends on what control plane software is
> taking
> > > > care
> > > > > of
> > > > > > your Kafka deployment. You probably have a layer that launches
> > Kafka
> > > > > > instances and monitors their health, right? If so, that layer
> > should
> > > > take
> > > > > > care of the mapping between instances and broker IDs and keep
> that
> > > in a
> > > > > > table persisted somewhere (e.g., DynamoDB).
> > > > > >
> > > > > > Eno
> > > > > >
> > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
> > cnu.text@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > EBS is one of the option. But we use instance level storage
> where
> > > we
> > > > > > loose
> > > > > > > all data as soon as we have a broker failed in AWS.
> > > > > > >
> > > > > > > In such scenario, anyone has better launch script or
> cofiguration
> > > can
> > > > > be
> > > > > > > executed on new broker to retain the old id not conflicting
> with
> > > > > existing
> > > > > > > broker ids.
> > > > > > >
> > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > > > andrey.dyachkov@gmail.com
> > > > > > > wrote:
> > > > > > >
> > > > > > > > You can attach EBS volume, which will store data and
> > > metadata(e.g.
> > > > > > broker
> > > > > > > > id), and then attach it to the new AWS instance and start
> > Kafka,
> > > it
> > > > > > will
> > > > > > > > pick the broker id plus you won’t need to rebalance the
> > cluster.
> > > > > > > >
> > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > > > nareshgoud.dulam@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > > > > >
> > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > > > cnu.text@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Kafka experts,
> > > > > > > > > >
> > > > > > > > > > We are running Kafka on AWS, main question is what is the
> > > best
> > > > > way
> > > > > > to
> > > > > > > > > > retain broker.id on new instance spun-up in-place of
> > > > > > instance/broker
> > > > > > > > > > failed.
> > > > > > > > > >
> > > > > > > > > > We are currently running Kafka in AWS with broker.id
> gets
> > > auto
> > > > > > > > > generated.
> > > > > > > > > > But we are having issues when a broker is failed, new
> > > > > > broker/instance
> > > > > > > > > > spun-up in AWS get assigned with new broker.id. The
> issue
> > > is,
> > > > > with
> > > > > > > > this
> > > > > > > > > > approach, we need to re-assign the topics/replications on
> > to
> > > > the
> > > > > > new
> > > > > > > > > broker
> > > > > > > > > > manually.
> > > > > > > > > >
> > > > > > > > > > We learned that, replication can be auto resolved by
> Kafka,
> > > if
> > > > we
> > > > > > can
> > > > > > > > > > manage to get the same broker.id on the new AWS instance
> > > > spun-up
> > > > > > > > > in-place
> > > > > > > > > > of failed broker/instance.
> > > > > > > > > >
> > > > > > > > > > I have read, we can set broker.id.generation.enable=
> false,
> > > but
> > > > > > what
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > best way to identify and retain the broker.id? Any
> > > links/help
> > > > is
> > > > > > > > > > appreciated.
> > > > > > > > > > Thanks and Regards,
> > > > > > > > > > Cnu
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Thanks,
> > > > > > > > > Naresh
> > > > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks,
> > > > > > > > Andrey
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Amit Pal
> > > >
> > >
> > >
> > > --
> > > Kaufman Ng
> > > +1 646 961 8063
> > > Solutions Architect | Confluent | www.confluent.io
> > >
> >
>


-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Ryanne Dolan <ry...@gmail.com>.
> if we are not using EBS

In that case, what's the point of keeping the broker id? The data will be
lost anyway right?


On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cn...@gmail.com> wrote:

> Yes, I understand we need to specify the required broker.id in
> server.properties/meta.properties file in-order to retain the broker.id
> which was failed.
>
> We can write custom script to query the zookeeper on launch process to get
> the broker.id needs to be set.
>
> Do we have any standards/script/way defined/preferred for doing this or
> suggested by Kafka experts if we are not using EBS?
>
> Thanks and Regards,
> Srinivas
>
> On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io> wrote:
>
> > Srinivas,
> >
> > You might want to check out the AWS best practices from this blog post:
> >
> >
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
> >
> > Kafka broker ids by default are auto-generated, but you can also specify
> > values (in server.properties file). By having static broker ids, it's
> > easier to recover in general. The blog post above mentions it in more
> > details.
> >
> > Good luck.
> >
> > On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:
> >
> > > If at all you were hell bent on doing this, you could use zookeeper to
> > find
> > > out the health of current brokers along with their broker id. That
> should
> > > help re-spin/start the unhealthy instance with same instance id.
> > >
> > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com>
> > > wrote:
> > >
> > > > having all this stored in DB is getting too complicated, especially
> > with
> > > > instance level storage and not EBS.
> > > >
> > > > I am sure there should be easy way to retain the old broker.id for
> new
> > > AWS
> > > > instance spun-up for auto replication.
> > > >
> > > > Any other ideas/help is appreciated.
> > > >
> > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <eno.thereska@gmail.com
> >
> > > > wrote:
> > > >
> > > > > The general answer depends on what control plane software is taking
> > > care
> > > > of
> > > > > your Kafka deployment. You probably have a layer that launches
> Kafka
> > > > > instances and monitors their health, right? If so, that layer
> should
> > > take
> > > > > care of the mapping between instances and broker IDs and keep that
> > in a
> > > > > table persisted somewhere (e.g., DynamoDB).
> > > > >
> > > > > Eno
> > > > >
> > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <
> cnu.text@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > EBS is one of the option. But we use instance level storage where
> > we
> > > > > loose
> > > > > > all data as soon as we have a broker failed in AWS.
> > > > > >
> > > > > > In such scenario, anyone has better launch script or cofiguration
> > can
> > > > be
> > > > > > executed on new broker to retain the old id not conflicting with
> > > > existing
> > > > > > broker ids.
> > > > > >
> > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > > andrey.dyachkov@gmail.com
> > > > > > wrote:
> > > > > >
> > > > > > > You can attach EBS volume, which will store data and
> > metadata(e.g.
> > > > > broker
> > > > > > > id), and then attach it to the new AWS instance and start
> Kafka,
> > it
> > > > > will
> > > > > > > pick the broker id plus you won’t need to rebalance the
> cluster.
> > > > > > >
> > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > > nareshgoud.dulam@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > > > >
> > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > > cnu.text@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Kafka experts,
> > > > > > > > >
> > > > > > > > > We are running Kafka on AWS, main question is what is the
> > best
> > > > way
> > > > > to
> > > > > > > > > retain broker.id on new instance spun-up in-place of
> > > > > instance/broker
> > > > > > > > > failed.
> > > > > > > > >
> > > > > > > > > We are currently running Kafka in AWS with broker.id gets
> > auto
> > > > > > > > generated.
> > > > > > > > > But we are having issues when a broker is failed, new
> > > > > broker/instance
> > > > > > > > > spun-up in AWS get assigned with new broker.id. The issue
> > is,
> > > > with
> > > > > > > this
> > > > > > > > > approach, we need to re-assign the topics/replications on
> to
> > > the
> > > > > new
> > > > > > > > broker
> > > > > > > > > manually.
> > > > > > > > >
> > > > > > > > > We learned that, replication can be auto resolved by Kafka,
> > if
> > > we
> > > > > can
> > > > > > > > > manage to get the same broker.id on the new AWS instance
> > > spun-up
> > > > > > > > in-place
> > > > > > > > > of failed broker/instance.
> > > > > > > > >
> > > > > > > > > I have read, we can set broker.id.generation.enable= false,
> > but
> > > > > what
> > > > > > is
> > > > > > > > the
> > > > > > > > > best way to identify and retain the broker.id? Any
> > links/help
> > > is
> > > > > > > > > appreciated.
> > > > > > > > > Thanks and Regards,
> > > > > > > > > Cnu
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Thanks,
> > > > > > > > Naresh
> > > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Andrey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Amit Pal
> > >
> >
> >
> > --
> > Kaufman Ng
> > +1 646 961 8063
> > Solutions Architect | Confluent | www.confluent.io
> >
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
Yes, I understand we need to specify the required broker.id in
server.properties/meta.properties file in-order to retain the broker.id
which was failed.

We can write custom script to query the zookeeper on launch process to get
the broker.id needs to be set.

Do we have any standards/script/way defined/preferred for doing this or
suggested by Kafka experts if we are not using EBS?

Thanks and Regards,
Srinivas

On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <ka...@confluent.io> wrote:

> Srinivas,
>
> You might want to check out the AWS best practices from this blog post:
>
> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
>
> Kafka broker ids by default are auto-generated, but you can also specify
> values (in server.properties file). By having static broker ids, it's
> easier to recover in general. The blog post above mentions it in more
> details.
>
> Good luck.
>
> On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:
>
> > If at all you were hell bent on doing this, you could use zookeeper to
> find
> > out the health of current brokers along with their broker id. That should
> > help re-spin/start the unhealthy instance with same instance id.
> >
> > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com>
> > wrote:
> >
> > > having all this stored in DB is getting too complicated, especially
> with
> > > instance level storage and not EBS.
> > >
> > > I am sure there should be easy way to retain the old broker.id for new
> > AWS
> > > instance spun-up for auto replication.
> > >
> > > Any other ideas/help is appreciated.
> > >
> > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <en...@gmail.com>
> > > wrote:
> > >
> > > > The general answer depends on what control plane software is taking
> > care
> > > of
> > > > your Kafka deployment. You probably have a layer that launches Kafka
> > > > instances and monitors their health, right? If so, that layer should
> > take
> > > > care of the mapping between instances and broker IDs and keep that
> in a
> > > > table persisted somewhere (e.g., DynamoDB).
> > > >
> > > > Eno
> > > >
> > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com>
> > > > wrote:
> > > >
> > > > > EBS is one of the option. But we use instance level storage where
> we
> > > > loose
> > > > > all data as soon as we have a broker failed in AWS.
> > > > >
> > > > > In such scenario, anyone has better launch script or cofiguration
> can
> > > be
> > > > > executed on new broker to retain the old id not conflicting with
> > > existing
> > > > > broker ids.
> > > > >
> > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > > andrey.dyachkov@gmail.com
> > > > > wrote:
> > > > >
> > > > > > You can attach EBS volume, which will store data and
> metadata(e.g.
> > > > broker
> > > > > > id), and then attach it to the new AWS instance and start Kafka,
> it
> > > > will
> > > > > > pick the broker id plus you won’t need to rebalance the cluster.
> > > > > >
> > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > > nareshgoud.dulam@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > > >
> > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > > cnu.text@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka experts,
> > > > > > > >
> > > > > > > > We are running Kafka on AWS, main question is what is the
> best
> > > way
> > > > to
> > > > > > > > retain broker.id on new instance spun-up in-place of
> > > > instance/broker
> > > > > > > > failed.
> > > > > > > >
> > > > > > > > We are currently running Kafka in AWS with broker.id gets
> auto
> > > > > > > generated.
> > > > > > > > But we are having issues when a broker is failed, new
> > > > broker/instance
> > > > > > > > spun-up in AWS get assigned with new broker.id. The issue
> is,
> > > with
> > > > > > this
> > > > > > > > approach, we need to re-assign the topics/replications on to
> > the
> > > > new
> > > > > > > broker
> > > > > > > > manually.
> > > > > > > >
> > > > > > > > We learned that, replication can be auto resolved by Kafka,
> if
> > we
> > > > can
> > > > > > > > manage to get the same broker.id on the new AWS instance
> > spun-up
> > > > > > > in-place
> > > > > > > > of failed broker/instance.
> > > > > > > >
> > > > > > > > I have read, we can set broker.id.generation.enable= false,
> but
> > > > what
> > > > > is
> > > > > > > the
> > > > > > > > best way to identify and retain the broker.id? Any
> links/help
> > is
> > > > > > > > appreciated.
> > > > > > > > Thanks and Regards,
> > > > > > > > Cnu
> > > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Naresh
> > > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > > http://hadoopandspark.blogspot.com/
> > > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Andrey
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Thanks and Regards,
> > Amit Pal
> >
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Kaufman Ng <ka...@confluent.io>.
Srinivas,

You might want to check out the AWS best practices from this blog post:
https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/

Kafka broker ids by default are auto-generated, but you can also specify
values (in server.properties file). By having static broker ids, it's
easier to recover in general. The blog post above mentions it in more
details.

Good luck.

On Thu, Nov 15, 2018 at 8:09 AM amit pal <am...@gmail.com> wrote:

> If at all you were hell bent on doing this, you could use zookeeper to find
> out the health of current brokers along with their broker id. That should
> help re-spin/start the unhealthy instance with same instance id.
>
> On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com>
> wrote:
>
> > having all this stored in DB is getting too complicated, especially with
> > instance level storage and not EBS.
> >
> > I am sure there should be easy way to retain the old broker.id for new
> AWS
> > instance spun-up for auto replication.
> >
> > Any other ideas/help is appreciated.
> >
> > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <en...@gmail.com>
> > wrote:
> >
> > > The general answer depends on what control plane software is taking
> care
> > of
> > > your Kafka deployment. You probably have a layer that launches Kafka
> > > instances and monitors their health, right? If so, that layer should
> take
> > > care of the mapping between instances and broker IDs and keep that in a
> > > table persisted somewhere (e.g., DynamoDB).
> > >
> > > Eno
> > >
> > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com>
> > > wrote:
> > >
> > > > EBS is one of the option. But we use instance level storage where we
> > > loose
> > > > all data as soon as we have a broker failed in AWS.
> > > >
> > > > In such scenario, anyone has better launch script or cofiguration can
> > be
> > > > executed on new broker to retain the old id not conflicting with
> > existing
> > > > broker ids.
> > > >
> > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > > andrey.dyachkov@gmail.com
> > > > wrote:
> > > >
> > > > > You can attach EBS volume, which will store data and metadata(e.g.
> > > broker
> > > > > id), and then attach it to the new AWS instance and start Kafka, it
> > > will
> > > > > pick the broker id plus you won’t need to rebalance the cluster.
> > > > >
> > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> > nareshgoud.dulam@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > > >
> > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> > cnu.text@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Kafka experts,
> > > > > > >
> > > > > > > We are running Kafka on AWS, main question is what is the best
> > way
> > > to
> > > > > > > retain broker.id on new instance spun-up in-place of
> > > instance/broker
> > > > > > > failed.
> > > > > > >
> > > > > > > We are currently running Kafka in AWS with broker.id gets auto
> > > > > > generated.
> > > > > > > But we are having issues when a broker is failed, new
> > > broker/instance
> > > > > > > spun-up in AWS get assigned with new broker.id. The issue is,
> > with
> > > > > this
> > > > > > > approach, we need to re-assign the topics/replications on to
> the
> > > new
> > > > > > broker
> > > > > > > manually.
> > > > > > >
> > > > > > > We learned that, replication can be auto resolved by Kafka, if
> we
> > > can
> > > > > > > manage to get the same broker.id on the new AWS instance
> spun-up
> > > > > > in-place
> > > > > > > of failed broker/instance.
> > > > > > >
> > > > > > > I have read, we can set broker.id.generation.enable= false, but
> > > what
> > > > is
> > > > > > the
> > > > > > > best way to identify and retain the broker.id? Any links/help
> is
> > > > > > > appreciated.
> > > > > > > Thanks and Regards,
> > > > > > > Cnu
> > > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Naresh
> > > > > > www.linkedin.com/in/naresh-dulam
> > > > > > http://hadoopandspark.blogspot.com/
> > > > > >
> > > > > --
> > > > > Thanks,
> > > > > Andrey
> > > > >
> > > >
> > >
> >
>
>
> --
> Thanks and Regards,
> Amit Pal
>


-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by amit pal <am...@gmail.com>.
If at all you were hell bent on doing this, you could use zookeeper to find
out the health of current brokers along with their broker id. That should
help re-spin/start the unhealthy instance with same instance id.

On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu <cn...@gmail.com> wrote:

> having all this stored in DB is getting too complicated, especially with
> instance level storage and not EBS.
>
> I am sure there should be easy way to retain the old broker.id for new AWS
> instance spun-up for auto replication.
>
> Any other ideas/help is appreciated.
>
> On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <en...@gmail.com>
> wrote:
>
> > The general answer depends on what control plane software is taking care
> of
> > your Kafka deployment. You probably have a layer that launches Kafka
> > instances and monitors their health, right? If so, that layer should take
> > care of the mapping between instances and broker IDs and keep that in a
> > table persisted somewhere (e.g., DynamoDB).
> >
> > Eno
> >
> > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com>
> > wrote:
> >
> > > EBS is one of the option. But we use instance level storage where we
> > loose
> > > all data as soon as we have a broker failed in AWS.
> > >
> > > In such scenario, anyone has better launch script or cofiguration can
> be
> > > executed on new broker to retain the old id not conflicting with
> existing
> > > broker ids.
> > >
> > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> > andrey.dyachkov@gmail.com
> > > wrote:
> > >
> > > > You can attach EBS volume, which will store data and metadata(e.g.
> > broker
> > > > id), and then attach it to the new AWS instance and start Kafka, it
> > will
> > > > pick the broker id plus you won’t need to rebalance the cluster.
> > > >
> > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <
> nareshgoud.dulam@gmail.com>
> > > > wrote:
> > > >
> > > > > Static IP. Buying static IP may help. I am not aws expert
> > > > >
> > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <
> cnu.text@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Kafka experts,
> > > > > >
> > > > > > We are running Kafka on AWS, main question is what is the best
> way
> > to
> > > > > > retain broker.id on new instance spun-up in-place of
> > instance/broker
> > > > > > failed.
> > > > > >
> > > > > > We are currently running Kafka in AWS with broker.id gets auto
> > > > > generated.
> > > > > > But we are having issues when a broker is failed, new
> > broker/instance
> > > > > > spun-up in AWS get assigned with new broker.id. The issue is,
> with
> > > > this
> > > > > > approach, we need to re-assign the topics/replications on to the
> > new
> > > > > broker
> > > > > > manually.
> > > > > >
> > > > > > We learned that, replication can be auto resolved by Kafka, if we
> > can
> > > > > > manage to get the same broker.id on the new AWS instance spun-up
> > > > > in-place
> > > > > > of failed broker/instance.
> > > > > >
> > > > > > I have read, we can set broker.id.generation.enable= false, but
> > what
> > > is
> > > > > the
> > > > > > best way to identify and retain the broker.id? Any links/help is
> > > > > > appreciated.
> > > > > > Thanks and Regards,
> > > > > > Cnu
> > > > > >
> > > > > --
> > > > > Thanks,
> > > > > Naresh
> > > > > www.linkedin.com/in/naresh-dulam
> > > > > http://hadoopandspark.blogspot.com/
> > > > >
> > > > --
> > > > Thanks,
> > > > Andrey
> > > >
> > >
> >
>


-- 
Thanks and Regards,
Amit Pal

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
having all this stored in DB is getting too complicated, especially with
instance level storage and not EBS.

I am sure there should be easy way to retain the old broker.id for new AWS
instance spun-up for auto replication.

Any other ideas/help is appreciated.

On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska <en...@gmail.com> wrote:

> The general answer depends on what control plane software is taking care of
> your Kafka deployment. You probably have a layer that launches Kafka
> instances and monitors their health, right? If so, that layer should take
> care of the mapping between instances and broker IDs and keep that in a
> table persisted somewhere (e.g., DynamoDB).
>
> Eno
>
> On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com>
> wrote:
>
> > EBS is one of the option. But we use instance level storage where we
> loose
> > all data as soon as we have a broker failed in AWS.
> >
> > In such scenario, anyone has better launch script or cofiguration can be
> > executed on new broker to retain the old id not conflicting with existing
> > broker ids.
> >
> > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <
> andrey.dyachkov@gmail.com
> > wrote:
> >
> > > You can attach EBS volume, which will store data and metadata(e.g.
> broker
> > > id), and then attach it to the new AWS instance and start Kafka, it
> will
> > > pick the broker id plus you won’t need to rebalance the cluster.
> > >
> > > On Wed 14. Nov 2018 at 19:48, naresh Goud <na...@gmail.com>
> > > wrote:
> > >
> > > > Static IP. Buying static IP may help. I am not aws expert
> > > >
> > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cnu.text@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hello Kafka experts,
> > > > >
> > > > > We are running Kafka on AWS, main question is what is the best way
> to
> > > > > retain broker.id on new instance spun-up in-place of
> instance/broker
> > > > > failed.
> > > > >
> > > > > We are currently running Kafka in AWS with broker.id gets auto
> > > > generated.
> > > > > But we are having issues when a broker is failed, new
> broker/instance
> > > > > spun-up in AWS get assigned with new broker.id. The issue is, with
> > > this
> > > > > approach, we need to re-assign the topics/replications on to the
> new
> > > > broker
> > > > > manually.
> > > > >
> > > > > We learned that, replication can be auto resolved by Kafka, if we
> can
> > > > > manage to get the same broker.id on the new AWS instance spun-up
> > > > in-place
> > > > > of failed broker/instance.
> > > > >
> > > > > I have read, we can set broker.id.generation.enable= false, but
> what
> > is
> > > > the
> > > > > best way to identify and retain the broker.id? Any links/help is
> > > > > appreciated.
> > > > > Thanks and Regards,
> > > > > Cnu
> > > > >
> > > > --
> > > > Thanks,
> > > > Naresh
> > > > www.linkedin.com/in/naresh-dulam
> > > > http://hadoopandspark.blogspot.com/
> > > >
> > > --
> > > Thanks,
> > > Andrey
> > >
> >
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Eno Thereska <en...@gmail.com>.
The general answer depends on what control plane software is taking care of
your Kafka deployment. You probably have a layer that launches Kafka
instances and monitors their health, right? If so, that layer should take
care of the mapping between instances and broker IDs and keep that in a
table persisted somewhere (e.g., DynamoDB).

Eno

On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cn...@gmail.com> wrote:

> EBS is one of the option. But we use instance level storage where we loose
> all data as soon as we have a broker failed in AWS.
>
> In such scenario, anyone has better launch script or cofiguration can be
> executed on new broker to retain the old id not conflicting with existing
> broker ids.
>
> On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <andrey.dyachkov@gmail.com
> wrote:
>
> > You can attach EBS volume, which will store data and metadata(e.g. broker
> > id), and then attach it to the new AWS instance and start Kafka, it will
> > pick the broker id plus you won’t need to rebalance the cluster.
> >
> > On Wed 14. Nov 2018 at 19:48, naresh Goud <na...@gmail.com>
> > wrote:
> >
> > > Static IP. Buying static IP may help. I am not aws expert
> > >
> > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com>
> > > wrote:
> > >
> > > > Hello Kafka experts,
> > > >
> > > > We are running Kafka on AWS, main question is what is the best way to
> > > > retain broker.id on new instance spun-up in-place of instance/broker
> > > > failed.
> > > >
> > > > We are currently running Kafka in AWS with broker.id gets auto
> > > generated.
> > > > But we are having issues when a broker is failed, new broker/instance
> > > > spun-up in AWS get assigned with new broker.id. The issue is, with
> > this
> > > > approach, we need to re-assign the topics/replications on to the new
> > > broker
> > > > manually.
> > > >
> > > > We learned that, replication can be auto resolved by Kafka, if we can
> > > > manage to get the same broker.id on the new AWS instance spun-up
> > > in-place
> > > > of failed broker/instance.
> > > >
> > > > I have read, we can set broker.id.generation.enable= false, but what
> is
> > > the
> > > > best way to identify and retain the broker.id? Any links/help is
> > > > appreciated.
> > > > Thanks and Regards,
> > > > Cnu
> > > >
> > > --
> > > Thanks,
> > > Naresh
> > > www.linkedin.com/in/naresh-dulam
> > > http://hadoopandspark.blogspot.com/
> > >
> > --
> > Thanks,
> > Andrey
> >
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
EBS is one of the option. But we use instance level storage where we loose
all data as soon as we have a broker failed in AWS.

In such scenario, anyone has better launch script or cofiguration can be
executed on new broker to retain the old id not conflicting with existing
broker ids.

On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <andrey.dyachkov@gmail.com
wrote:

> You can attach EBS volume, which will store data and metadata(e.g. broker
> id), and then attach it to the new AWS instance and start Kafka, it will
> pick the broker id plus you won’t need to rebalance the cluster.
>
> On Wed 14. Nov 2018 at 19:48, naresh Goud <na...@gmail.com>
> wrote:
>
> > Static IP. Buying static IP may help. I am not aws expert
> >
> > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com>
> > wrote:
> >
> > > Hello Kafka experts,
> > >
> > > We are running Kafka on AWS, main question is what is the best way to
> > > retain broker.id on new instance spun-up in-place of instance/broker
> > > failed.
> > >
> > > We are currently running Kafka in AWS with broker.id gets auto
> > generated.
> > > But we are having issues when a broker is failed, new broker/instance
> > > spun-up in AWS get assigned with new broker.id. The issue is, with
> this
> > > approach, we need to re-assign the topics/replications on to the new
> > broker
> > > manually.
> > >
> > > We learned that, replication can be auto resolved by Kafka, if we can
> > > manage to get the same broker.id on the new AWS instance spun-up
> > in-place
> > > of failed broker/instance.
> > >
> > > I have read, we can set broker.id.generation.enable= false, but what is
> > the
> > > best way to identify and retain the broker.id? Any links/help is
> > > appreciated.
> > > Thanks and Regards,
> > > Cnu
> > >
> > --
> > Thanks,
> > Naresh
> > www.linkedin.com/in/naresh-dulam
> > http://hadoopandspark.blogspot.com/
> >
> --
> Thanks,
> Andrey
>

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by amit pal <am...@gmail.com>.
Wow, that seems like an anti-pattern.
Replication itself should be enough to resurrect the cluster in case of
node failures. Technically, you shouldn't have to maintain the same broker
id. There must be something else going on with replication.

On Thu, Nov 15, 2018 at 12:28 AM Andrey Dyachkov <an...@gmail.com>
wrote:

> You can attach EBS volume, which will store data and metadata(e.g. broker
> id), and then attach it to the new AWS instance and start Kafka, it will
> pick the broker id plus you won’t need to rebalance the cluster.
>
> On Wed 14. Nov 2018 at 19:48, naresh Goud <na...@gmail.com>
> wrote:
>
> > Static IP. Buying static IP may help. I am not aws expert
> >
> > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com>
> > wrote:
> >
> > > Hello Kafka experts,
> > >
> > > We are running Kafka on AWS, main question is what is the best way to
> > > retain broker.id on new instance spun-up in-place of instance/broker
> > > failed.
> > >
> > > We are currently running Kafka in AWS with broker.id gets auto
> > generated.
> > > But we are having issues when a broker is failed, new broker/instance
> > > spun-up in AWS get assigned with new broker.id. The issue is, with
> this
> > > approach, we need to re-assign the topics/replications on to the new
> > broker
> > > manually.
> > >
> > > We learned that, replication can be auto resolved by Kafka, if we can
> > > manage to get the same broker.id on the new AWS instance spun-up
> > in-place
> > > of failed broker/instance.
> > >
> > > I have read, we can set broker.id.generation.enable= false, but what is
> > the
> > > best way to identify and retain the broker.id? Any links/help is
> > > appreciated.
> > > Thanks and Regards,
> > > Cnu
> > >
> > --
> > Thanks,
> > Naresh
> > www.linkedin.com/in/naresh-dulam
> > http://hadoopandspark.blogspot.com/
> >
> --
> Thanks,
> Andrey
>


-- 
Thanks and Regards,
Amit Pal

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Andrey Dyachkov <an...@gmail.com>.
You can attach EBS volume, which will store data and metadata(e.g. broker
id), and then attach it to the new AWS instance and start Kafka, it will
pick the broker id plus you won’t need to rebalance the cluster.

On Wed 14. Nov 2018 at 19:48, naresh Goud <na...@gmail.com>
wrote:

> Static IP. Buying static IP may help. I am not aws expert
>
> On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com>
> wrote:
>
> > Hello Kafka experts,
> >
> > We are running Kafka on AWS, main question is what is the best way to
> > retain broker.id on new instance spun-up in-place of instance/broker
> > failed.
> >
> > We are currently running Kafka in AWS with broker.id gets auto
> generated.
> > But we are having issues when a broker is failed, new broker/instance
> > spun-up in AWS get assigned with new broker.id. The issue is, with this
> > approach, we need to re-assign the topics/replications on to the new
> broker
> > manually.
> >
> > We learned that, replication can be auto resolved by Kafka, if we can
> > manage to get the same broker.id on the new AWS instance spun-up
> in-place
> > of failed broker/instance.
> >
> > I have read, we can set broker.id.generation.enable= false, but what is
> the
> > best way to identify and retain the broker.id? Any links/help is
> > appreciated.
> > Thanks and Regards,
> > Cnu
> >
> --
> Thanks,
> Naresh
> www.linkedin.com/in/naresh-dulam
> http://hadoopandspark.blogspot.com/
>
-- 
Thanks,
Andrey

Re: Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed

Posted by Srinivas Rapolu <cn...@gmail.com>.
thanks Naresh for the quick response. But we don't want to make use of any
elastic IP in this case.

I found something like, we can manually get the broker-id using some script
mentioned  @ http://tech.gc.com/scaling-with-kafka/ while instance is
getting launched.

Trying to find, if there is any other/better way to
re-assingn/retain/impersonate the old broker.id.

Thanks and Regards,
Srinivas

On Wed, Nov 14, 2018 at 11:48 AM naresh Goud <na...@gmail.com>
wrote:

> Static IP. Buying static IP may help. I am not aws expert
>
> On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cn...@gmail.com>
> wrote:
>
> > Hello Kafka experts,
> >
> > We are running Kafka on AWS, main question is what is the best way to
> > retain broker.id on new instance spun-up in-place of instance/broker
> > failed.
> >
> > We are currently running Kafka in AWS with broker.id gets auto
> generated.
> > But we are having issues when a broker is failed, new broker/instance
> > spun-up in AWS get assigned with new broker.id. The issue is, with this
> > approach, we need to re-assign the topics/replications on to the new
> broker
> > manually.
> >
> > We learned that, replication can be auto resolved by Kafka, if we can
> > manage to get the same broker.id on the new AWS instance spun-up
> in-place
> > of failed broker/instance.
> >
> > I have read, we can set broker.id.generation.enable= false, but what is
> the
> > best way to identify and retain the broker.id? Any links/help is
> > appreciated.
> > Thanks and Regards,
> > Cnu
> >
> --
> Thanks,
> Naresh
> www.linkedin.com/in/naresh-dulam
> http://hadoopandspark.blogspot.com/
>