You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Balthasar Schopman <b....@tech.leaseweb.com> on 2015/12/10 16:29:04 UTC
Creating partition with many partitions
Hi,
We are considering an architecture with a Kafka cluster of 3 nodes and a high number of consumers. We see that with a low number of partitions, e.g. 3, and a higher number of consumers, e.g. 16, there will be only 3 consumers actually consuming data, because only the owners of partitions can consume messages. To see the owners we do the following:
$ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group consumer_group
Group Topic Pid Offset logSize Lag Owner
consumer_group statistics 0 5335 5373 38 consumer_group_balthasar-1449651803301-63a1d620-0
consumer_group statistics 1 5335 5374 39 consumer_group_balthasar-1449651803820-35a84426-0
consumer_group statistics 2 5335 5374 39 consumer_group_balthasar-1449651803934-2b3cc1bd-0
One solution to being able to have many consumers is to increase the amount of partitions to a high number, e.g. 1024. This would put more load on the machines running Kafka, but would this load be crazy? The machines that'll be running Kafka have 64GB RAM and a Xeon E5-2620 CPU (6 cores clocked at 2GHz, 24 hardware threads in total).
Are there any other reasons not to use such a high number of partitions?
Kind regards,
Balthasar Schopman
Kind regards,
Balthasar Schopman
Software Developer
LeaseWeb Technologies B.V.
T: +31 20 316 0232
M:
E: b.schopman@tech.leaseweb.com
W: http://www.leaseweb.com
Luttenbergweg 8, 1101 EC Amsterdam, Netherlands
Re: Creating partition with many partitions
Posted by Ofir Manor <of...@equalum.io>.
Hi,
I think this is the best reference on the actual trade-off (more/less
partitions):
http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
As a guess, for a three-node Kafka cluster, I don't see how useful would be
to have 1024 consumers in a single consumer group, just for read
parallelism sake within that group.
Anyway, maybe test and measure the actual throughput you get on your setup
- or even better, set your throughput and availability goals first
(replication levels, throughput if a node fails etc) and then tune the
configuration accordingly (including number of partitions) to support it.
You may need to add disks or nodes to get there, depending on your actual
goals.
Ofir
On Thu, Dec 10, 2015 at 5:29 PM, Balthasar Schopman <
b.schopman@tech.leaseweb.com> wrote:
> Hi,
>
> We are considering an architecture with a Kafka cluster of 3 nodes and a
> high number of consumers. We see that with a low number of partitions, e.g.
> 3, and a higher number of consumers, e.g. 16, there will be only 3
> consumers actually consuming data, because only the owners of partitions
> can consume messages. To see the owners we do the following:
>
> $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> localhost:2181 --group consumer_group
> Group Topic Pid Offset logSize Lag Owner
> consumer_group statistics 0 5335 5373 38
> consumer_group_balthasar-1449651803301-63a1d620-0
> consumer_group statistics 1 5335 5374 39
> consumer_group_balthasar-1449651803820-35a84426-0
> consumer_group statistics 2 5335 5374 39
> consumer_group_balthasar-1449651803934-2b3cc1bd-0
>
> One solution to being able to have many consumers is to increase the
> amount of partitions to a high number, e.g. 1024. This would put more load
> on the machines running Kafka, but would this load be crazy? The machines
> that'll be running Kafka have 64GB RAM and a Xeon E5-2620 CPU (6 cores
> clocked at 2GHz, 24 hardware threads in total).
>
> Are there any other reasons not to use such a high number of partitions?
>
> Kind regards,
> Balthasar Schopman
>
> Kind regards,
>
> Balthasar Schopman
> Software Developer
> LeaseWeb Technologies B.V.
>
> T: +31 20 316 0232
> M:
> E: b.schopman@tech.leaseweb.com
> W: http://www.leaseweb.com
>
> Luttenbergweg 8, 1101 EC Amsterdam, Netherlands
>
>
>