Dynamic partitioning

[István <le...@gmail.com>]

Hi all,

My understanding is that with 0.8.1.x you can manually change the number of
partitions on the broker, and this change is going to be picked up by the
producers and consumers (high level).

kafka-topics.sh --alter --zookeeper zk.net:2181/stream --topic test
--partitions 3

Is that the case?

Is there a way to adjust the number of partitions based on the load? This
might not be the best way of scaling up and down (auto-scaling) Kafka, so
if there is a better way I would like to know how.

Thank you in advance,
Istvan


-- 
the sun shines for all

Re: Dynamic partitioning

[Joe Stein <jo...@stealth.ly>]

That command will change how many partitions the topic has.

What you are looking for I think is
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
which allows you to change what partitions are running on which replicas
and which replicas are the preferred leader... some more documentation on
that https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
(that entire section https://kafka.apache.org/documentation.html#basic_ops
has the type of information are you are looking for I think)

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Fri, Sep 12, 2014 at 11:33 AM, István <le...@gmail.com> wrote:

> Hi all,
>
> My understanding is that with 0.8.1.x you can manually change the number of
> partitions on the broker, and this change is going to be picked up by the
> producers and consumers (high level).
>
> kafka-topics.sh --alter --zookeeper zk.net:2181/stream --topic test
> --partitions 3
>
> Is that the case?
>
> Is there a way to adjust the number of partitions based on the load? This
> might not be the best way of scaling up and down (auto-scaling) Kafka, so
> if there is a better way I would like to know how.
>
> Thank you in advance,
> Istvan
>
>
> --
> the sun shines for all
>