You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Amit K <am...@gmail.com> on 2016/07/19 01:55:18 UTC

Regarding kafka partition and replication

Hi,

I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 zookeeper
cluster. So total 9 brokers spread across 3 different machines. I am
adhered to Kafka 0.9.

In order to optimally use the infrastructure for 2 topics (as of now, is
not expected to grow drastically in near future), I am thinking of having 9
partitions with 3 (or 6?) replication factor. Will this help me having good
distribution of partitions and replicas across brokers? System does not
have hugh load (<50 requests/sec of less than 1 kb load each) as of now and
is neither expected to get higher load than this.

If this replication and partition does not help, please suggest a better
topic partition and replication strategy.

Also please guide me with any articles or documents about setting up multi
node Kafka cluster in regard to partition, replication, general properties
to be used (a kind of good practice etc.)

Thanks,
Amit

Re: Regarding kafka partition and replication

Posted by Amit K <am...@gmail.com>.
Thanks Dave,

Seems good option!

Thanks,
Amit

On Wed, Jul 20, 2016 at 10:24 PM, Tauzell, Dave <
Dave.Tauzell@surescripts.com> wrote:

> For fault -tolerance you want the replicated partitions to be on different
> physical servers.  I think a better set is to have 3 brokers, 9 partitions
> and a replication factor of 2.  This will ensure that the replicated data
> is always on a different physical hosts.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Amit K [mailto:amitk.abk@gmail.com]
> Sent: Wednesday, July 20, 2016 11:40 AM
> To: users@kafka.apache.org
> Subject: Re: Regarding kafka partition and replication
>
> Thanks for reply.
>
> The infrastructure, which I mentioned, is already in place and is getting
> used so can not alter it.
>
> But given that, if I have 9 partitions with replication factor of 2, will
> that help me to have a good fault tolerant and optimal, in regard with
> hardware use, system?
>
> Thanks,
> Amit
>
> On Tue, Jul 19, 2016 at 6:48 PM, Tauzell, Dave <
> Dave.Tauzell@surescripts.com
> > wrote:
>
> > Having multiple brokers on the same node has a couple of problems for
> > a production installation:
> >
> > 1. You'll have multiple brokers contending for disk and memory
> > resources 2. You could have your partitions replicated to the same
> > node which means if that node fails you would lose data.
> >
> > I think you are better off having 3 nodes with 3 brokers.   You can keep
> > with 9 partitions in case you want to add physical nodes in the future
> > and have a replication factor of 2 or 3.
> >
> > -Dave
> >
> > Dave Tauzell | Senior Software Engineer | Surescripts
> > O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
> > Connect with us: Twitter I LinkedIn I Facebook I YouTube
> >
> >
> > -----Original Message-----
> > From: Amit K [mailto:amitk.abk@gmail.com]
> > Sent: Monday, July 18, 2016 8:55 PM
> > To: users@kafka.apache.org
> > Subject: Regarding kafka partition and replication
> >
> > Hi,
> >
> > I have 3 nodes, each with 3 brokers, Kafka cluster along with 3
> > zookeeper cluster. So total 9 brokers spread across 3 different
> > machines. I am adhered to Kafka 0.9.
> >
> > In order to optimally use the infrastructure for 2 topics (as of now,
> > is not expected to grow drastically in near future), I am thinking of
> > having 9 partitions with 3 (or 6?) replication factor. Will this help
> > me having good distribution of partitions and replicas across brokers?
> > System does not have hugh load (<50 requests/sec of less than 1 kb
> > load each) as of now and is neither expected to get higher load than
> this.
> >
> > If this replication and partition does not help, please suggest a
> > better topic partition and replication strategy.
> >
> > Also please guide me with any articles or documents about setting up
> > multi node Kafka cluster in regard to partition, replication, general
> > properties to be used (a kind of good practice etc.)
> >
> > Thanks,
> > Amit
> > This e-mail and any files transmitted with it are confidential, may
> > contain sensitive information, and are intended solely for the use of
> > the individual or entity to whom they are addressed. If you have
> > received this e-mail in error, please notify the sender by reply
> > e-mail immediately and destroy all copies of the e-mail and any
> attachments.
> >
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>

RE: Regarding kafka partition and replication

Posted by "Tauzell, Dave" <Da...@surescripts.com>.
For fault -tolerance you want the replicated partitions to be on different physical servers.  I think a better set is to have 3 brokers, 9 partitions and a replication factor of 2.  This will ensure that the replicated data is always on a different physical hosts.

-Dave

Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube


-----Original Message-----
From: Amit K [mailto:amitk.abk@gmail.com]
Sent: Wednesday, July 20, 2016 11:40 AM
To: users@kafka.apache.org
Subject: Re: Regarding kafka partition and replication

Thanks for reply.

The infrastructure, which I mentioned, is already in place and is getting used so can not alter it.

But given that, if I have 9 partitions with replication factor of 2, will that help me to have a good fault tolerant and optimal, in regard with hardware use, system?

Thanks,
Amit

On Tue, Jul 19, 2016 at 6:48 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
> wrote:

> Having multiple brokers on the same node has a couple of problems for
> a production installation:
>
> 1. You'll have multiple brokers contending for disk and memory
> resources 2. You could have your partitions replicated to the same
> node which means if that node fails you would lose data.
>
> I think you are better off having 3 nodes with 3 brokers.   You can keep
> with 9 partitions in case you want to add physical nodes in the future
> and have a replication factor of 2 or 3.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Amit K [mailto:amitk.abk@gmail.com]
> Sent: Monday, July 18, 2016 8:55 PM
> To: users@kafka.apache.org
> Subject: Regarding kafka partition and replication
>
> Hi,
>
> I have 3 nodes, each with 3 brokers, Kafka cluster along with 3
> zookeeper cluster. So total 9 brokers spread across 3 different
> machines. I am adhered to Kafka 0.9.
>
> In order to optimally use the infrastructure for 2 topics (as of now,
> is not expected to grow drastically in near future), I am thinking of
> having 9 partitions with 3 (or 6?) replication factor. Will this help
> me having good distribution of partitions and replicas across brokers?
> System does not have hugh load (<50 requests/sec of less than 1 kb
> load each) as of now and is neither expected to get higher load than this.
>
> If this replication and partition does not help, please suggest a
> better topic partition and replication strategy.
>
> Also please guide me with any articles or documents about setting up
> multi node Kafka cluster in regard to partition, replication, general
> properties to be used (a kind of good practice etc.)
>
> Thanks,
> Amit
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of
> the individual or entity to whom they are addressed. If you have
> received this e-mail in error, please notify the sender by reply
> e-mail immediately and destroy all copies of the e-mail and any attachments.
>
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Regarding kafka partition and replication

Posted by Amit K <am...@gmail.com>.
Thanks for reply.

The infrastructure, which I mentioned, is already in place and is getting
used so can not alter it.

But given that, if I have 9 partitions with replication factor of 2, will
that help me to have a good fault tolerant and optimal, in regard with
hardware use, system?

Thanks,
Amit

On Tue, Jul 19, 2016 at 6:48 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
> wrote:

> Having multiple brokers on the same node has a couple of problems for a
> production installation:
>
> 1. You'll have multiple brokers contending for disk and memory resources
> 2. You could have your partitions replicated to the same node which means
> if that node fails you would lose data.
>
> I think you are better off having 3 nodes with 3 brokers.   You can keep
> with 9 partitions in case you want to add physical nodes in the future and
> have a replication factor of 2 or 3.
>
> -Dave
>
> Dave Tauzell | Senior Software Engineer | Surescripts
> O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
> Connect with us: Twitter I LinkedIn I Facebook I YouTube
>
>
> -----Original Message-----
> From: Amit K [mailto:amitk.abk@gmail.com]
> Sent: Monday, July 18, 2016 8:55 PM
> To: users@kafka.apache.org
> Subject: Regarding kafka partition and replication
>
> Hi,
>
> I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 zookeeper
> cluster. So total 9 brokers spread across 3 different machines. I am
> adhered to Kafka 0.9.
>
> In order to optimally use the infrastructure for 2 topics (as of now, is
> not expected to grow drastically in near future), I am thinking of having 9
> partitions with 3 (or 6?) replication factor. Will this help me having good
> distribution of partitions and replicas across brokers? System does not
> have hugh load (<50 requests/sec of less than 1 kb load each) as of now and
> is neither expected to get higher load than this.
>
> If this replication and partition does not help, please suggest a better
> topic partition and replication strategy.
>
> Also please guide me with any articles or documents about setting up multi
> node Kafka cluster in regard to partition, replication, general properties
> to be used (a kind of good practice etc.)
>
> Thanks,
> Amit
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>

RE: Regarding kafka partition and replication

Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Having multiple brokers on the same node has a couple of problems for a production installation:

1. You'll have multiple brokers contending for disk and memory resources
2. You could have your partitions replicated to the same node which means if that node fails you would lose data.

I think you are better off having 3 nodes with 3 brokers.   You can keep with 9 partitions in case you want to add physical nodes in the future and have a replication factor of 2 or 3.

-Dave

Dave Tauzell | Senior Software Engineer | Surescripts
O: 651.855.3042 | www.surescripts.com |   Dave.Tauzell@surescripts.com
Connect with us: Twitter I LinkedIn I Facebook I YouTube


-----Original Message-----
From: Amit K [mailto:amitk.abk@gmail.com]
Sent: Monday, July 18, 2016 8:55 PM
To: users@kafka.apache.org
Subject: Regarding kafka partition and replication

Hi,

I have 3 nodes, each with 3 brokers, Kafka cluster along with 3 zookeeper cluster. So total 9 brokers spread across 3 different machines. I am adhered to Kafka 0.9.

In order to optimally use the infrastructure for 2 topics (as of now, is not expected to grow drastically in near future), I am thinking of having 9 partitions with 3 (or 6?) replication factor. Will this help me having good distribution of partitions and replicas across brokers? System does not have hugh load (<50 requests/sec of less than 1 kb load each) as of now and is neither expected to get higher load than this.

If this replication and partition does not help, please suggest a better topic partition and replication strategy.

Also please guide me with any articles or documents about setting up multi node Kafka cluster in regard to partition, replication, general properties to be used (a kind of good practice etc.)

Thanks,
Amit
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.