How to change number of partition topic wise on the fly

[Tarun Kumar <ag...@gmail.com>]

Is there any way to change the number of partitions topic wise on the fly
without restarting kafka server?

Re: How to change number of partition topic wise on the fly

[R S <my...@gmail.com>]

Other way could be to create multiple topics , as topics can be created on
the fly. Producer can send data in a round robin fashion to individual
topic as in load balanced way.  Only problem i can see one might end up
having  lots of topics say 50-100 . Is this an issue??

-rks

On Tue, May 15, 2012 at 10:51 PM, Edward Smith <es...@stardotstar.org>wrote:

> Speaking from inexperience:
>
> Wouldn't it be possible to just set the number of partitions on the
> broker to a high number, and then use your own partitioner logic to
> always select partition 0.  Then, when you want more partitions,
> modify the partitioner/producer to use more partitions?  This way you
> only have to restart producers and not the brokers.
>
> I don't actually understand entirely why the broker needs to know how
> many partitions there will be.  It seems he could just honor requests
> to write to arbitrary partition numbers.
>
> Ed
>
> On Tue, May 15, 2012 at 12:22 PM, Jun Rao <ju...@gmail.com> wrote:
> > Tarun,
> >
> > In 0.8, we plan to have an admin DDL that supports changing #partitions
> of
> > a topic and it won't require broker restarts.
> >
> > Jun
> >
> > On Tue, May 15, 2012 at 9:13 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> >wrote:
> >
> >> Currently, I have one producer publishing messages on one topic (say
> t1). I
> >> have configured number of partitions on broker as 3 for topic t1 in
> broker
> >> config. At Consumer side, i have one consumer group with 3 consumers. I
> >> have mapped these 3 consumers to 3 partitions (one to one mapping), each
> >> consumer creates one map task and i store consumer's offsets in hdfs.
> >>
> >> But in future, if suppose producer starts producing messages at higher
> >> speed, I would like to increase number of partitions for this topic (t1)
> >> and number of consumers also for that consumer group. I want to avoid
> >> broker restart. I was wondering if there is any way to do that. Broker
> >> restart can be problematic in case i have multiple brokers.
> >>
> >> On Tue, May 15, 2012 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
> >>
> >> > No, this is a config change on the broker and requires a broker
> restart.
> >> >
> >> > Jun
> >> >
> >> > On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <
> agrawal.tarun23@gmail.com
> >> > >wrote:
> >> >
> >> > > Is there any way to change the number of partitions topic wise on
> the
> >> fly
> >> > > without restarting kafka server?
> >> > >
> >> >
> >>
>

Re: How to change number of partition topic wise on the fly

[Edward Smith <es...@stardotstar.org>]

Speaking from inexperience:

Wouldn't it be possible to just set the number of partitions on the
broker to a high number, and then use your own partitioner logic to
always select partition 0.  Then, when you want more partitions,
modify the partitioner/producer to use more partitions?  This way you
only have to restart producers and not the brokers.

I don't actually understand entirely why the broker needs to know how
many partitions there will be.  It seems he could just honor requests
to write to arbitrary partition numbers.

Ed

On Tue, May 15, 2012 at 12:22 PM, Jun Rao <ju...@gmail.com> wrote:
> Tarun,
>
> In 0.8, we plan to have an admin DDL that supports changing #partitions of
> a topic and it won't require broker restarts.
>
> Jun
>
> On Tue, May 15, 2012 at 9:13 AM, Tarun Kumar <ag...@gmail.com>wrote:
>
>> Currently, I have one producer publishing messages on one topic (say t1). I
>> have configured number of partitions on broker as 3 for topic t1 in broker
>> config. At Consumer side, i have one consumer group with 3 consumers. I
>> have mapped these 3 consumers to 3 partitions (one to one mapping), each
>> consumer creates one map task and i store consumer's offsets in hdfs.
>>
>> But in future, if suppose producer starts producing messages at higher
>> speed, I would like to increase number of partitions for this topic (t1)
>> and number of consumers also for that consumer group. I want to avoid
>> broker restart. I was wondering if there is any way to do that. Broker
>> restart can be problematic in case i have multiple brokers.
>>
>> On Tue, May 15, 2012 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
>>
>> > No, this is a config change on the broker and requires a broker restart.
>> >
>> > Jun
>> >
>> > On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <agrawal.tarun23@gmail.com
>> > >wrote:
>> >
>> > > Is there any way to change the number of partitions topic wise on the
>> fly
>> > > without restarting kafka server?
>> > >
>> >
>>

Re: How to change number of partition topic wise on the fly

[Jun Rao <ju...@gmail.com>]

Tarun,

In 0.8, we plan to have an admin DDL that supports changing #partitions of
a topic and it won't require broker restarts.

Jun

On Tue, May 15, 2012 at 9:13 AM, Tarun Kumar <ag...@gmail.com>wrote:

> Currently, I have one producer publishing messages on one topic (say t1). I
> have configured number of partitions on broker as 3 for topic t1 in broker
> config. At Consumer side, i have one consumer group with 3 consumers. I
> have mapped these 3 consumers to 3 partitions (one to one mapping), each
> consumer creates one map task and i store consumer's offsets in hdfs.
>
> But in future, if suppose producer starts producing messages at higher
> speed, I would like to increase number of partitions for this topic (t1)
> and number of consumers also for that consumer group. I want to avoid
> broker restart. I was wondering if there is any way to do that. Broker
> restart can be problematic in case i have multiple brokers.
>
> On Tue, May 15, 2012 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > No, this is a config change on the broker and requires a broker restart.
> >
> > Jun
> >
> > On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> > >wrote:
> >
> > > Is there any way to change the number of partitions topic wise on the
> fly
> > > without restarting kafka server?
> > >
> >
>

Re: How to change number of partition topic wise on the fly

[Tarun Kumar <ag...@gmail.com>]

Currently, I have one producer publishing messages on one topic (say t1). I
have configured number of partitions on broker as 3 for topic t1 in broker
config. At Consumer side, i have one consumer group with 3 consumers. I
have mapped these 3 consumers to 3 partitions (one to one mapping), each
consumer creates one map task and i store consumer's offsets in hdfs.

But in future, if suppose producer starts producing messages at higher
speed, I would like to increase number of partitions for this topic (t1)
and number of consumers also for that consumer group. I want to avoid
broker restart. I was wondering if there is any way to do that. Broker
restart can be problematic in case i have multiple brokers.

On Tue, May 15, 2012 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:

> No, this is a config change on the broker and requires a broker restart.
>
> Jun
>
> On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <agrawal.tarun23@gmail.com
> >wrote:
>
> > Is there any way to change the number of partitions topic wise on the fly
> > without restarting kafka server?
> >
>

Re: How to change number of partition topic wise on the fly

[Jun Rao <ju...@gmail.com>]

No, this is a config change on the broker and requires a broker restart.

Jun

On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <ag...@gmail.com>wrote:

> Is there any way to change the number of partitions topic wise on the fly
> without restarting kafka server?
>