You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2014/11/03 23:03:20 UTC
Dynamically adding Kafka brokers
Hi,
How do people handle situations, and specifically the broker.id property,
where the Kafka (broker) cluster is not fully defined right away?
Here's the use case we have at Sematext:
* Our software ships as a VM
* All components run in this single VM, including 1 Kafka broker
* Of course, this is just for a nice OOTB experience, but to scale one
needs to have more instances of this VM, including more Kafka brokers
* *One can clone our VM and launch N instances of it, but because we have a
single Kafka broker config with a single broker.id <http://broker.id> in
it, one can't just launch more of these VMs and expect to see more Kafka
brokers join the cluster. One would have to change the broker.id
<http://broker.id> on each new VM instance.*
How do others handle this in a software that is packages and ships to user
and is not in your direct control to allow you to edit configs?
Would it be best to have a script that connect to ZooKeeper to get the list
of all existing brokers and their IDs and then generate a new distinct ID +
config for the new Kafka broker?
Or are there slicker ways to do this that people use?
Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Dynamically adding Kafka brokers
Posted by Neha Narkhede <ne...@gmail.com>.
I agree that KAFKA-1070 would be great to get in. I especially felt the
need for something like this while using a few other systems that automated
the port, id etc to give a good OOTB experience. Sorry, I lost track of the
review. Will do so in the next few days.
Thanks,
Neha
On Mon, Nov 3, 2014 at 3:56 PM, Jay Kreps <ja...@gmail.com> wrote:
> I agree it would be really nice to get KAFKA-1070 figured out.
>
> FWIW, the reason for having a name or id other than ip was to make it
> possible to move the identity to another physical server (e.g. scp the data
> directory) and have it perform the same role on that new piece of hardware.
> Systems that tie the data to a particular host tend to be sort of hated on
> since you can't do anything simple/stupid to back them up or replace them.
>
> -Jay
>
> On Mon, Nov 3, 2014 at 2:23 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > +1
> > Thats what we use to generate broker id in automatic deployments.
> > This method makes troubleshooting easier (you know where each broker is
> > running), and doesn't require keeping extra files around.
> >
> > On Mon, Nov 3, 2014 at 2:17 PM, Joe Stein <jo...@stealth.ly> wrote:
> >
> > > Most folks strip the IP and use that as the broker.id. KAFKA-1070 does
> > not
> > > yet accommodate for that very widely used method. I think it would be
> bad
> > > if KAFKA-1070 only worked for new installations because that is how
> > people
> > > use Kafka today (per
> > >
> > >
> >
> https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
> > > )
> > >
> > > On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jj...@gmail.com>
> wrote:
> > >
> > > > KAFKA-1070 will help with this and is pending a review.
> > > >
> > > > On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > > > > Hi,
> > > > >
> > > > > How do people handle situations, and specifically the broker.id
> > > > property,
> > > > > where the Kafka (broker) cluster is not fully defined right away?
> > > > >
> > > > > Here's the use case we have at Sematext:
> > > > > * Our software ships as a VM
> > > > > * All components run in this single VM, including 1 Kafka broker
> > > > > * Of course, this is just for a nice OOTB experience, but to scale
> > one
> > > > > needs to have more instances of this VM, including more Kafka
> brokers
> > > > > * *One can clone our VM and launch N instances of it, but because
> we
> > > > have a
> > > > > single Kafka broker config with a single broker.id <
> http://broker.id
> > >
> > > in
> > > > > it, one can't just launch more of these VMs and expect to see more
> > > Kafka
> > > > > brokers join the cluster. One would have to change the broker.id
> > > > > <http://broker.id> on each new VM instance.*
> > > > >
> > > > > How do others handle this in a software that is packages and ships
> to
> > > > user
> > > > > and is not in your direct control to allow you to edit configs?
> > > > >
> > > > > Would it be best to have a script that connect to ZooKeeper to get
> > the
> > > > list
> > > > > of all existing brokers and their IDs and then generate a new
> > distinct
> > > > ID +
> > > > > config for the new Kafka broker?
> > > > >
> > > > > Or are there slicker ways to do this that people use?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > >
> > > >
> > >
> >
>
Re: Dynamically adding Kafka brokers
Posted by Jay Kreps <ja...@gmail.com>.
I agree it would be really nice to get KAFKA-1070 figured out.
FWIW, the reason for having a name or id other than ip was to make it
possible to move the identity to another physical server (e.g. scp the data
directory) and have it perform the same role on that new piece of hardware.
Systems that tie the data to a particular host tend to be sort of hated on
since you can't do anything simple/stupid to back them up or replace them.
-Jay
On Mon, Nov 3, 2014 at 2:23 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> +1
> Thats what we use to generate broker id in automatic deployments.
> This method makes troubleshooting easier (you know where each broker is
> running), and doesn't require keeping extra files around.
>
> On Mon, Nov 3, 2014 at 2:17 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Most folks strip the IP and use that as the broker.id. KAFKA-1070 does
> not
> > yet accommodate for that very widely used method. I think it would be bad
> > if KAFKA-1070 only worked for new installations because that is how
> people
> > use Kafka today (per
> >
> >
> https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
> > )
> >
> > On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jj...@gmail.com> wrote:
> >
> > > KAFKA-1070 will help with this and is pending a review.
> > >
> > > On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > > > Hi,
> > > >
> > > > How do people handle situations, and specifically the broker.id
> > > property,
> > > > where the Kafka (broker) cluster is not fully defined right away?
> > > >
> > > > Here's the use case we have at Sematext:
> > > > * Our software ships as a VM
> > > > * All components run in this single VM, including 1 Kafka broker
> > > > * Of course, this is just for a nice OOTB experience, but to scale
> one
> > > > needs to have more instances of this VM, including more Kafka brokers
> > > > * *One can clone our VM and launch N instances of it, but because we
> > > have a
> > > > single Kafka broker config with a single broker.id <http://broker.id
> >
> > in
> > > > it, one can't just launch more of these VMs and expect to see more
> > Kafka
> > > > brokers join the cluster. One would have to change the broker.id
> > > > <http://broker.id> on each new VM instance.*
> > > >
> > > > How do others handle this in a software that is packages and ships to
> > > user
> > > > and is not in your direct control to allow you to edit configs?
> > > >
> > > > Would it be best to have a script that connect to ZooKeeper to get
> the
> > > list
> > > > of all existing brokers and their IDs and then generate a new
> distinct
> > > ID +
> > > > config for the new Kafka broker?
> > > >
> > > > Or are there slicker ways to do this that people use?
> > > >
> > > > Thanks,
> > > > Otis
> > > > --
> > > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> >
>
Re: Dynamically adding Kafka brokers
Posted by Gwen Shapira <gs...@cloudera.com>.
+1
Thats what we use to generate broker id in automatic deployments.
This method makes troubleshooting easier (you know where each broker is
running), and doesn't require keeping extra files around.
On Mon, Nov 3, 2014 at 2:17 PM, Joe Stein <jo...@stealth.ly> wrote:
> Most folks strip the IP and use that as the broker.id. KAFKA-1070 does not
> yet accommodate for that very widely used method. I think it would be bad
> if KAFKA-1070 only worked for new installations because that is how people
> use Kafka today (per
>
> https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
> )
>
> On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> > KAFKA-1070 will help with this and is pending a review.
> >
> > On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > > Hi,
> > >
> > > How do people handle situations, and specifically the broker.id
> > property,
> > > where the Kafka (broker) cluster is not fully defined right away?
> > >
> > > Here's the use case we have at Sematext:
> > > * Our software ships as a VM
> > > * All components run in this single VM, including 1 Kafka broker
> > > * Of course, this is just for a nice OOTB experience, but to scale one
> > > needs to have more instances of this VM, including more Kafka brokers
> > > * *One can clone our VM and launch N instances of it, but because we
> > have a
> > > single Kafka broker config with a single broker.id <http://broker.id>
> in
> > > it, one can't just launch more of these VMs and expect to see more
> Kafka
> > > brokers join the cluster. One would have to change the broker.id
> > > <http://broker.id> on each new VM instance.*
> > >
> > > How do others handle this in a software that is packages and ships to
> > user
> > > and is not in your direct control to allow you to edit configs?
> > >
> > > Would it be best to have a script that connect to ZooKeeper to get the
> > list
> > > of all existing brokers and their IDs and then generate a new distinct
> > ID +
> > > config for the new Kafka broker?
> > >
> > > Or are there slicker ways to do this that people use?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>
Re: Dynamically adding Kafka brokers
Posted by Joe Stein <jo...@stealth.ly>.
Most folks strip the IP and use that as the broker.id. KAFKA-1070 does not
yet accommodate for that very widely used method. I think it would be bad
if KAFKA-1070 only worked for new installations because that is how people
use Kafka today (per
https://issues.apache.org/jira/browse/KAFKA-1070?focusedCommentId=14085808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085808
)
On Mon, Nov 3, 2014 at 2:12 PM, Joel Koshy <jj...@gmail.com> wrote:
> KAFKA-1070 will help with this and is pending a review.
>
> On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> > Hi,
> >
> > How do people handle situations, and specifically the broker.id
> property,
> > where the Kafka (broker) cluster is not fully defined right away?
> >
> > Here's the use case we have at Sematext:
> > * Our software ships as a VM
> > * All components run in this single VM, including 1 Kafka broker
> > * Of course, this is just for a nice OOTB experience, but to scale one
> > needs to have more instances of this VM, including more Kafka brokers
> > * *One can clone our VM and launch N instances of it, but because we
> have a
> > single Kafka broker config with a single broker.id <http://broker.id> in
> > it, one can't just launch more of these VMs and expect to see more Kafka
> > brokers join the cluster. One would have to change the broker.id
> > <http://broker.id> on each new VM instance.*
> >
> > How do others handle this in a software that is packages and ships to
> user
> > and is not in your direct control to allow you to edit configs?
> >
> > Would it be best to have a script that connect to ZooKeeper to get the
> list
> > of all existing brokers and their IDs and then generate a new distinct
> ID +
> > config for the new Kafka broker?
> >
> > Or are there slicker ways to do this that people use?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
>
>
Re: Dynamically adding Kafka brokers
Posted by Joel Koshy <jj...@gmail.com>.
KAFKA-1070 will help with this and is pending a review.
On Mon, Nov 03, 2014 at 05:03:20PM -0500, Otis Gospodnetic wrote:
> Hi,
>
> How do people handle situations, and specifically the broker.id property,
> where the Kafka (broker) cluster is not fully defined right away?
>
> Here's the use case we have at Sematext:
> * Our software ships as a VM
> * All components run in this single VM, including 1 Kafka broker
> * Of course, this is just for a nice OOTB experience, but to scale one
> needs to have more instances of this VM, including more Kafka brokers
> * *One can clone our VM and launch N instances of it, but because we have a
> single Kafka broker config with a single broker.id <http://broker.id> in
> it, one can't just launch more of these VMs and expect to see more Kafka
> brokers join the cluster. One would have to change the broker.id
> <http://broker.id> on each new VM instance.*
>
> How do others handle this in a software that is packages and ships to user
> and is not in your direct control to allow you to edit configs?
>
> Would it be best to have a script that connect to ZooKeeper to get the list
> of all existing brokers and their IDs and then generate a new distinct ID +
> config for the new Kafka broker?
>
> Or are there slicker ways to do this that people use?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
Re: Dynamically adding Kafka brokers
Posted by Florian Dambrine <fl...@gumgum.com>.
I am automating Kafka clusters in the cloud and I am using Ips (stripped to
keep only numbers) to define the broker ids.
You might have to add a shell script or whatever fits to set the broker id.
I hope this might help a bit.
Regards,
Le 3 nov. 2014 23:04, "Otis Gospodnetic" <ot...@gmail.com> a
écrit :
> Hi,
>
> How do people handle situations, and specifically the broker.id property,
> where the Kafka (broker) cluster is not fully defined right away?
>
> Here's the use case we have at Sematext:
> * Our software ships as a VM
> * All components run in this single VM, including 1 Kafka broker
> * Of course, this is just for a nice OOTB experience, but to scale one
> needs to have more instances of this VM, including more Kafka brokers
> * *One can clone our VM and launch N instances of it, but because we have a
> single Kafka broker config with a single broker.id <http://broker.id> in
> it, one can't just launch more of these VMs and expect to see more Kafka
> brokers join the cluster. One would have to change the broker.id
> <http://broker.id> on each new VM instance.*
>
> How do others handle this in a software that is packages and ships to user
> and is not in your direct control to allow you to edit configs?
>
> Would it be best to have a script that connect to ZooKeeper to get the list
> of all existing brokers and their IDs and then generate a new distinct ID +
> config for the new Kafka broker?
>
> Or are there slicker ways to do this that people use?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>