You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Skip Montanaro <sk...@gmail.com> on 2017/11/28 14:06:03 UTC
Multiple brokers - do they share the load?
Apologies for the rather long set-up...
I've been using Kafka as a client for a few months now. The setup I've
been using has three brokers on separate servers, all listening to
port 9092. My consumers always connected to server1:9092. I've ignored
server2 and server3.
Now I'm starting to mess around a bit with setting up my own
itsy-bitsy cluster. Step one is a single instance at host1:9092. Next
step in the instructions (I'm following the recipe laid out in the
documentation) will be to add two more brokers at host1:9093 and
host1:9094.
My question: If every consumer connects to host1:9092 will the brokers
listening to the other ports starve for attention, or does the
connection process somehow redirect clients to the other brokers so
the three (or more) brokers get fairly equitable loads?
Thanks,
Skip Montanaro
Re: Multiple brokers - do they share the load?
Posted by Svante Karlsson <sv...@csi.se>.
You are connecting to a single seed node - your kafka library will then
under the hood connect to the partition leaders for each partition you
subscribe or post to.
The load is not different compared to if you gave all nodes as connect
parameter. However if your seed node crashes then your client cannot
connect to the cluster.,.
2017-11-28 15:06 GMT+01:00 Skip Montanaro <sk...@gmail.com>:
> Apologies for the rather long set-up...
>
> I've been using Kafka as a client for a few months now. The setup I've
> been using has three brokers on separate servers, all listening to
> port 9092. My consumers always connected to server1:9092. I've ignored
> server2 and server3.
>
> Now I'm starting to mess around a bit with setting up my own
> itsy-bitsy cluster. Step one is a single instance at host1:9092. Next
> step in the instructions (I'm following the recipe laid out in the
> documentation) will be to add two more brokers at host1:9093 and
> host1:9094.
>
> My question: If every consumer connects to host1:9092 will the brokers
> listening to the other ports starve for attention, or does the
> connection process somehow redirect clients to the other brokers so
> the three (or more) brokers get fairly equitable loads?
>
> Thanks,
>
> Skip Montanaro
>
Re: Multiple brokers - do they share the load?
Posted by Skip Montanaro <sk...@gmail.com>.
Ben wrote:
> If you set these configs at broker start they will automatically replicate
> newly created topics:
>
> default.replication.factor
> num.partitions=3
Thanks. (I assume the first config is missing a trailing "=3" bit).
Final question (I think). If I restart my brokers with those
configuration changes, will existing topics acquire the desired
replication factor, or will I need to do something else to tweak
existing topics?
Skip
> Ben
>
> On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro <sk...@gmail.com>
> wrote:
>
>> Are dynamically created topics thus never replicated?
>>
>> On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <Da...@surescripts.com>
>> wrote:
>>
>> > You then also need to set this up for each topic you create:
>> >
>> > > bin/kafka-topics.sh --create --zookeeper localhost:2181
>> > --replication-factor 3 --partitions 3 --topic my-replicated-topic
>> >
>> > -Dave
>> >
>> > -----Original Message-----
>> > From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
>> > Sent: Thursday, November 30, 2017 4:20 PM
>> > To: users@kafka.apache.org
>> > Subject: Re: Multiple brokers - do they share the load?
>> >
>> > > If you create a partitioned topic with at least 3 partitions then you
>> > will see your client connect to all of the brokers. The client decides
>> > which partition a message should go to and then sends it directly to the
>> > broker that is the leader for that partition. If you have replicated
>> > topics, then the brokers themselves will also be connected to one-another
>> > in order to replication messages.
>> >
>> > Thanks. So if I create three brokers all pointing to the same Zookeeper
>> > instance, then define
>> >
>> > offsets.topic.replication.factor=3
>> > transaction.state.log.replication.factor=3
>> >
>> > the brokers will find each other via Zookeeper and dynamically created
>> > topics (which is all I've used so far) will be properly replicated?
>> > Should I also bump transaction.state.log.min.isr above 1?
>> >
>> > Apologies for such basic questions. I'm still trying to absorb the
>> > documentation related to overall architecture and broker specifics.
>> >
>> > Skip
>> >
>> > This e-mail and any files transmitted with it are confidential, may
>> > contain sensitive information, and are intended solely for the use of the
>> > individual or entity to whom they are addressed. If you have received
>> this
>> > e-mail in error, please notify the sender by reply e-mail immediately and
>> > destroy all copies of the e-mail and any attachments.
>> >
>>
>
> --
>
>
> This email, including attachments, is private and confidential. If you have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
> 7digital Group plc. Registered office: 69 Wilson Street, London EC2A 2BB.
> Registered in England and Wales. Registered No. 04843573.
Re: Multiple brokers - do they share the load?
Posted by Ben Davison <be...@7digital.com>.
If you set these configs at broker start they will automatically replicate
newly created topics:
default.replication.factor
num.partitions=3
Ben
On Fri, Dec 1, 2017 at 1:41 PM Skip Montanaro <sk...@gmail.com>
wrote:
> Are dynamically created topics thus never replicated?
>
> On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <Da...@surescripts.com>
> wrote:
>
> > You then also need to set this up for each topic you create:
> >
> > > bin/kafka-topics.sh --create --zookeeper localhost:2181
> > --replication-factor 3 --partitions 3 --topic my-replicated-topic
> >
> > -Dave
> >
> > -----Original Message-----
> > From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
> > Sent: Thursday, November 30, 2017 4:20 PM
> > To: users@kafka.apache.org
> > Subject: Re: Multiple brokers - do they share the load?
> >
> > > If you create a partitioned topic with at least 3 partitions then you
> > will see your client connect to all of the brokers. The client decides
> > which partition a message should go to and then sends it directly to the
> > broker that is the leader for that partition. If you have replicated
> > topics, then the brokers themselves will also be connected to one-another
> > in order to replication messages.
> >
> > Thanks. So if I create three brokers all pointing to the same Zookeeper
> > instance, then define
> >
> > offsets.topic.replication.factor=3
> > transaction.state.log.replication.factor=3
> >
> > the brokers will find each other via Zookeeper and dynamically created
> > topics (which is all I've used so far) will be properly replicated?
> > Should I also bump transaction.state.log.min.isr above 1?
> >
> > Apologies for such basic questions. I'm still trying to absorb the
> > documentation related to overall architecture and broker specifics.
> >
> > Skip
> >
> > This e-mail and any files transmitted with it are confidential, may
> > contain sensitive information, and are intended solely for the use of the
> > individual or entity to whom they are addressed. If you have received
> this
> > e-mail in error, please notify the sender by reply e-mail immediately and
> > destroy all copies of the e-mail and any attachments.
> >
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Group plc. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
RE: Multiple brokers - do they share the load?
Posted by Skip Montanaro <sk...@gmail.com>.
Are dynamically created topics thus never replicated?
On Nov 30, 2017 6:52 PM, "Tauzell, Dave" <Da...@surescripts.com>
wrote:
> You then also need to set this up for each topic you create:
>
> > bin/kafka-topics.sh --create --zookeeper localhost:2181
> --replication-factor 3 --partitions 3 --topic my-replicated-topic
>
> -Dave
>
> -----Original Message-----
> From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
> Sent: Thursday, November 30, 2017 4:20 PM
> To: users@kafka.apache.org
> Subject: Re: Multiple brokers - do they share the load?
>
> > If you create a partitioned topic with at least 3 partitions then you
> will see your client connect to all of the brokers. The client decides
> which partition a message should go to and then sends it directly to the
> broker that is the leader for that partition. If you have replicated
> topics, then the brokers themselves will also be connected to one-another
> in order to replication messages.
>
> Thanks. So if I create three brokers all pointing to the same Zookeeper
> instance, then define
>
> offsets.topic.replication.factor=3
> transaction.state.log.replication.factor=3
>
> the brokers will find each other via Zookeeper and dynamically created
> topics (which is all I've used so far) will be properly replicated?
> Should I also bump transaction.state.log.min.isr above 1?
>
> Apologies for such basic questions. I'm still trying to absorb the
> documentation related to overall architecture and broker specifics.
>
> Skip
>
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
RE: Multiple brokers - do they share the load?
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
You then also need to set this up for each topic you create:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic my-replicated-topic
-Dave
-----Original Message-----
From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
Sent: Thursday, November 30, 2017 4:20 PM
To: users@kafka.apache.org
Subject: Re: Multiple brokers - do they share the load?
> If you create a partitioned topic with at least 3 partitions then you will see your client connect to all of the brokers. The client decides which partition a message should go to and then sends it directly to the broker that is the leader for that partition. If you have replicated topics, then the brokers themselves will also be connected to one-another in order to replication messages.
Thanks. So if I create three brokers all pointing to the same Zookeeper instance, then define
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
the brokers will find each other via Zookeeper and dynamically created topics (which is all I've used so far) will be properly replicated?
Should I also bump transaction.state.log.min.isr above 1?
Apologies for such basic questions. I'm still trying to absorb the documentation related to overall architecture and broker specifics.
Skip
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
Re: Multiple brokers - do they share the load?
Posted by Skip Montanaro <sk...@gmail.com>.
> If you create a partitioned topic with at least 3 partitions then you will see your client connect to all of the brokers. The client decides which partition a message should go to and then sends it directly to the broker that is the leader for that partition. If you have replicated topics, then the brokers themselves will also be connected to one-another in order to replication messages.
Thanks. So if I create three brokers all pointing to the same
Zookeeper instance, then define
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
the brokers will find each other via Zookeeper and dynamically created
topics (which is all I've used so far) will be properly replicated?
Should I also bump transaction.state.log.min.isr above 1?
Apologies for such basic questions. I'm still trying to absorb the
documentation related to overall architecture and broker specifics.
Skip
RE: Multiple brokers - do they share the load?
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
If you create a partitioned topic with at least 3 partitions then you will see your client connect to all of the brokers. The client decides which partition a message should go to and then sends it directly to the broker that is the leader for that partition. If you have replicated topics, then the brokers themselves will also be connected to one-another in order to replication messages.
-Dave
-----Original Message-----
From: Skip Montanaro [mailto:skip.montanaro@gmail.com]
Sent: Tuesday, November 28, 2017 8:06 AM
To: users@kafka.apache.org
Subject: Multiple brokers - do they share the load?
Apologies for the rather long set-up...
I've been using Kafka as a client for a few months now. The setup I've been using has three brokers on separate servers, all listening to port 9092. My consumers always connected to server1:9092. I've ignored
server2 and server3.
Now I'm starting to mess around a bit with setting up my own itsy-bitsy cluster. Step one is a single instance at host1:9092. Next step in the instructions (I'm following the recipe laid out in the
documentation) will be to add two more brokers at host1:9093 and host1:9094.
My question: If every consumer connects to host1:9092 will the brokers listening to the other ports starve for attention, or does the connection process somehow redirect clients to the other brokers so the three (or more) brokers get fairly equitable loads?
Thanks,
Skip Montanaro
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.