You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sean Morris (semorris)" <se...@cisco.com> on 2016/02/16 15:08:09 UTC
Replication Factor and number of brokers
Should your number of brokers be atleast one more then your replication factor of your topic(s)?
So if I have a replication factor of 2, I need atleast 3 brokers?
Thanks,
Sean
Re: Replication Factor and number of brokers
Posted by Alex Loddengaard <al...@confluent.io>.
Thanks, Alexis and Ben!
Alex
On Wed, Feb 17, 2016 at 5:57 AM, Ben Stopford <be...@confluent.io> wrote:
> If you create a topic with more replicas than brokers it should throw an
> error but if you lose a broker you'd have under replicated partitions.
>
> B
>
> On Tuesday, 16 February 2016, Alex Loddengaard <al...@confluent.io> wrote:
>
> > Hi Sean, you'll want equal or more brokers than your replication factor.
> > Meaning, if your replication factor is 3, you'll want 3 or more brokers.
> >
> > I'm not sure what Kafka will do if you have fewer brokers than your
> > replication factor. It will either give you the highest replication
> factor
> > it can (in this case, the number of brokers), or it will put more than
> one
> > replica on some brokers. My guess is the former, but again, I'm not sure.
> >
> > Hope this helps.
> >
> > Alex
> >
> > On Tue, Feb 16, 2016 at 7:47 AM, Damian Guy <damian.guy@gmail.com
> > <javascript:;>> wrote:
> >
> > > Then you'll have under-replicated partitions. However, even if you
> have 3
> > > brokers with a replication factor of 2 and you lose a single broker
> > you'll
> > > still likely have under-replicated partitions.
> > > Partitions are assigned to brokers, 1 broker will be the leader and n
> > > brokers will be followers. If any of the brokers with replicas of the
> > > partition on it crash then you'll have under-replicated partitions.
> > >
> > >
> > > On 16 February 2016 at 14:45, Sean Morris (semorris) <
> semorris@cisco.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > So if I have a replication factor of 2, but only 2 brokers, then
> > > > replication works, but what if I lose one broker?
> > > >
> > > > Thanks,
> > > > Sean
> > > >
> > > > On 2/16/16, 9:14 AM, "Damian Guy" <damian.guy@gmail.com
> <javascript:;>>
> > wrote:
> > > >
> > > > >Hi,
> > > > >
> > > > >You need to have at least replication factor brokers.
> > > > >replication factor = 1 is no replication.
> > > > >
> > > > >HTH,
> > > > >Damian
> > > > >
> > > > >On 16 February 2016 at 14:08, Sean Morris (semorris) <
> > > semorris@cisco.com <javascript:;>>
> > > > >wrote:
> > > > >
> > > > >> Should your number of brokers be atleast one more then your
> > > replication
> > > > >> factor of your topic(s)?
> > > > >>
> > > > >> So if I have a replication factor of 2, I need atleast 3 brokers?
> > > > >>
> > > > >> Thanks,
> > > > >> Sean
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Alex Loddengaard | **Solutions Architect | Confluent*
> > *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> > <http://www.confluent.io/download>*
> >
>
--
*Alex Loddengaard | **Solutions Architect | Confluent*
*Download Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
Re: Replication Factor and number of brokers
Posted by Ben Stopford <be...@confluent.io>.
If you create a topic with more replicas than brokers it should throw an
error but if you lose a broker you'd have under replicated partitions.
B
On Tuesday, 16 February 2016, Alex Loddengaard <al...@confluent.io> wrote:
> Hi Sean, you'll want equal or more brokers than your replication factor.
> Meaning, if your replication factor is 3, you'll want 3 or more brokers.
>
> I'm not sure what Kafka will do if you have fewer brokers than your
> replication factor. It will either give you the highest replication factor
> it can (in this case, the number of brokers), or it will put more than one
> replica on some brokers. My guess is the former, but again, I'm not sure.
>
> Hope this helps.
>
> Alex
>
> On Tue, Feb 16, 2016 at 7:47 AM, Damian Guy <damian.guy@gmail.com
> <javascript:;>> wrote:
>
> > Then you'll have under-replicated partitions. However, even if you have 3
> > brokers with a replication factor of 2 and you lose a single broker
> you'll
> > still likely have under-replicated partitions.
> > Partitions are assigned to brokers, 1 broker will be the leader and n
> > brokers will be followers. If any of the brokers with replicas of the
> > partition on it crash then you'll have under-replicated partitions.
> >
> >
> > On 16 February 2016 at 14:45, Sean Morris (semorris) <semorris@cisco.com
> <javascript:;>>
> > wrote:
> >
> > > So if I have a replication factor of 2, but only 2 brokers, then
> > > replication works, but what if I lose one broker?
> > >
> > > Thanks,
> > > Sean
> > >
> > > On 2/16/16, 9:14 AM, "Damian Guy" <damian.guy@gmail.com <javascript:;>>
> wrote:
> > >
> > > >Hi,
> > > >
> > > >You need to have at least replication factor brokers.
> > > >replication factor = 1 is no replication.
> > > >
> > > >HTH,
> > > >Damian
> > > >
> > > >On 16 February 2016 at 14:08, Sean Morris (semorris) <
> > semorris@cisco.com <javascript:;>>
> > > >wrote:
> > > >
> > > >> Should your number of brokers be atleast one more then your
> > replication
> > > >> factor of your topic(s)?
> > > >>
> > > >> So if I have a replication factor of 2, I need atleast 3 brokers?
> > > >>
> > > >> Thanks,
> > > >> Sean
> > > >>
> > > >>
> > >
> > >
> >
>
>
>
> --
> *Alex Loddengaard | **Solutions Architect | Confluent*
> *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> <http://www.confluent.io/download>*
>
Re: Replication Factor and number of brokers
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
it will throw an exception:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala#L76-L78
On Tue, Feb 16, 2016 at 3:55 PM Alex Loddengaard <al...@confluent.io> wrote:
> Hi Sean, you'll want equal or more brokers than your replication factor.
> Meaning, if your replication factor is 3, you'll want 3 or more brokers.
>
> I'm not sure what Kafka will do if you have fewer brokers than your
> replication factor. It will either give you the highest replication factor
> it can (in this case, the number of brokers), or it will put more than one
> replica on some brokers. My guess is the former, but again, I'm not sure.
>
> Hope this helps.
>
> Alex
>
> On Tue, Feb 16, 2016 at 7:47 AM, Damian Guy <da...@gmail.com> wrote:
>
> > Then you'll have under-replicated partitions. However, even if you have 3
> > brokers with a replication factor of 2 and you lose a single broker
> you'll
> > still likely have under-replicated partitions.
> > Partitions are assigned to brokers, 1 broker will be the leader and n
> > brokers will be followers. If any of the brokers with replicas of the
> > partition on it crash then you'll have under-replicated partitions.
> >
> >
> > On 16 February 2016 at 14:45, Sean Morris (semorris) <semorris@cisco.com
> >
> > wrote:
> >
> > > So if I have a replication factor of 2, but only 2 brokers, then
> > > replication works, but what if I lose one broker?
> > >
> > > Thanks,
> > > Sean
> > >
> > > On 2/16/16, 9:14 AM, "Damian Guy" <da...@gmail.com> wrote:
> > >
> > > >Hi,
> > > >
> > > >You need to have at least replication factor brokers.
> > > >replication factor = 1 is no replication.
> > > >
> > > >HTH,
> > > >Damian
> > > >
> > > >On 16 February 2016 at 14:08, Sean Morris (semorris) <
> > semorris@cisco.com>
> > > >wrote:
> > > >
> > > >> Should your number of brokers be atleast one more then your
> > replication
> > > >> factor of your topic(s)?
> > > >>
> > > >> So if I have a replication factor of 2, I need atleast 3 brokers?
> > > >>
> > > >> Thanks,
> > > >> Sean
> > > >>
> > > >>
> > >
> > >
> >
>
>
>
> --
> *Alex Loddengaard | **Solutions Architect | Confluent*
> *Download Apache Kafka and Confluent Platform: www.confluent.io/download
> <http://www.confluent.io/download>*
>
Re: Replication Factor and number of brokers
Posted by Alex Loddengaard <al...@confluent.io>.
Hi Sean, you'll want equal or more brokers than your replication factor.
Meaning, if your replication factor is 3, you'll want 3 or more brokers.
I'm not sure what Kafka will do if you have fewer brokers than your
replication factor. It will either give you the highest replication factor
it can (in this case, the number of brokers), or it will put more than one
replica on some brokers. My guess is the former, but again, I'm not sure.
Hope this helps.
Alex
On Tue, Feb 16, 2016 at 7:47 AM, Damian Guy <da...@gmail.com> wrote:
> Then you'll have under-replicated partitions. However, even if you have 3
> brokers with a replication factor of 2 and you lose a single broker you'll
> still likely have under-replicated partitions.
> Partitions are assigned to brokers, 1 broker will be the leader and n
> brokers will be followers. If any of the brokers with replicas of the
> partition on it crash then you'll have under-replicated partitions.
>
>
> On 16 February 2016 at 14:45, Sean Morris (semorris) <se...@cisco.com>
> wrote:
>
> > So if I have a replication factor of 2, but only 2 brokers, then
> > replication works, but what if I lose one broker?
> >
> > Thanks,
> > Sean
> >
> > On 2/16/16, 9:14 AM, "Damian Guy" <da...@gmail.com> wrote:
> >
> > >Hi,
> > >
> > >You need to have at least replication factor brokers.
> > >replication factor = 1 is no replication.
> > >
> > >HTH,
> > >Damian
> > >
> > >On 16 February 2016 at 14:08, Sean Morris (semorris) <
> semorris@cisco.com>
> > >wrote:
> > >
> > >> Should your number of brokers be atleast one more then your
> replication
> > >> factor of your topic(s)?
> > >>
> > >> So if I have a replication factor of 2, I need atleast 3 brokers?
> > >>
> > >> Thanks,
> > >> Sean
> > >>
> > >>
> >
> >
>
--
*Alex Loddengaard | **Solutions Architect | Confluent*
*Download Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*
Re: Replication Factor and number of brokers
Posted by Damian Guy <da...@gmail.com>.
Then you'll have under-replicated partitions. However, even if you have 3
brokers with a replication factor of 2 and you lose a single broker you'll
still likely have under-replicated partitions.
Partitions are assigned to brokers, 1 broker will be the leader and n
brokers will be followers. If any of the brokers with replicas of the
partition on it crash then you'll have under-replicated partitions.
On 16 February 2016 at 14:45, Sean Morris (semorris) <se...@cisco.com>
wrote:
> So if I have a replication factor of 2, but only 2 brokers, then
> replication works, but what if I lose one broker?
>
> Thanks,
> Sean
>
> On 2/16/16, 9:14 AM, "Damian Guy" <da...@gmail.com> wrote:
>
> >Hi,
> >
> >You need to have at least replication factor brokers.
> >replication factor = 1 is no replication.
> >
> >HTH,
> >Damian
> >
> >On 16 February 2016 at 14:08, Sean Morris (semorris) <se...@cisco.com>
> >wrote:
> >
> >> Should your number of brokers be atleast one more then your replication
> >> factor of your topic(s)?
> >>
> >> So if I have a replication factor of 2, I need atleast 3 brokers?
> >>
> >> Thanks,
> >> Sean
> >>
> >>
>
>
Re: Replication Factor and number of brokers
Posted by "Sean Morris (semorris)" <se...@cisco.com>.
So if I have a replication factor of 2, but only 2 brokers, then
replication works, but what if I lose one broker?
Thanks,
Sean
On 2/16/16, 9:14 AM, "Damian Guy" <da...@gmail.com> wrote:
>Hi,
>
>You need to have at least replication factor brokers.
>replication factor = 1 is no replication.
>
>HTH,
>Damian
>
>On 16 February 2016 at 14:08, Sean Morris (semorris) <se...@cisco.com>
>wrote:
>
>> Should your number of brokers be atleast one more then your replication
>> factor of your topic(s)?
>>
>> So if I have a replication factor of 2, I need atleast 3 brokers?
>>
>> Thanks,
>> Sean
>>
>>
Re: Replication Factor and number of brokers
Posted by Damian Guy <da...@gmail.com>.
Hi,
You need to have at least replication factor brokers.
replication factor = 1 is no replication.
HTH,
Damian
On 16 February 2016 at 14:08, Sean Morris (semorris) <se...@cisco.com>
wrote:
> Should your number of brokers be atleast one more then your replication
> factor of your topic(s)?
>
> So if I have a replication factor of 2, I need atleast 3 brokers?
>
> Thanks,
> Sean
>
>