Dynamic configuration for partitions

[Evan Chan <ev...@ooyala.com>]

So currently if we want to bump the # of partitions, one needs to restart
Kafka with a new server.properties file.
Is there any way to make this dynamic?

Let's say that my load has increased 2x recently, it would be nice not to
have to restart Kafka.


By the way, the default partitioner has a bug in it in 0.6, the hash
function can return a negative value and throws an exception.

-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Re: Dynamic configuration for partitions

[Joel Koshy <jj...@gmail.com>]

Hi Evan,

Since the config values need to be in memory, I don't think it is
possible to avoid a restart if you want to change the partition count.
Avoiding a restart would mean exposing some kind of limited "hup"
command in kafka brokers to reread the configs on demand, or to reread
the configs on a schedule - I'm not sure if it is worth the effort
since restarts are quick unless the broker does not shut down cleanly.
In any case, If you are using the high-level (zookeeper-based)
producers/consumers you can avoid "outages" by doing a rolling restart
of your brokers.

Thanks for pointing out the default partitioner issue - not sure if we
need a jira for it. Can one of the committers review this:

diff --git a/core/src/main/scala/kafka/producer/DefaultPartitioner.scala
b/core/src/main/scala/kafka/producer/DefaultPartitioner.scala
index e1fac32..3459224 100644
--- a/core/src/main/scala/kafka/producer/DefaultPartitioner.scala
+++ b/core/src/main/scala/kafka/producer/DefaultPartitioner.scala
@@ -24,6 +24,6 @@ private[kafka] class DefaultPartitioner[T] extends
Partitioner[T] {
     if(key == null)
       random.nextInt(numPartitions)
     else
-      key.hashCode % numPartitions
+      math.abs(key.hashCode) % numPartitions
   }
 }

Thanks,

Joel

On Thu, Sep 15, 2011 at 11:30 AM, Evan Chan <ev...@ooyala.com> wrote:
> So currently if we want to bump the # of partitions, one needs to restart
> Kafka with a new server.properties file.
> Is there any way to make this dynamic?
>
> Let's say that my load has increased 2x recently, it would be nice not to
> have to restart Kafka.
>
>
> By the way, the default partitioner has a bug in it in 0.6, the hash
> function can return a negative value and throws an exception.
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>