You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Calvin Lei <ck...@gmail.com> on 2013/07/04 17:11:34 UTC
Changing the number of partitions after a topic is created
Hi I have a few topics created with 1 partition. After running the cluster
for a few days, I want to increase the partition to 10 to improve the
consumer throughput. I learnt that it is not supported in 0.8.
What is the recommendation of improving consumer throughput after a topic
is created and the data volume increased?
Regards,
Cal
Re: Changing the number of partitions after a topic is created
Posted by Jun Rao <ju...@gmail.com>.
We are shooting for providing a patch in a couple of weeks.
Thanks,
Jun
On Mon, Jul 8, 2013 at 10:33 AM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Jun,
>
> I wonder when will the tool be available? We're very interested in changing
> the number of partitions for a topic after creation too.
>
> Thanks!
>
> Tim
>
>
> On Thu, Jul 4, 2013 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Currently, once a topic is created, the number of partitions can't be
> > changed. We are working on a tool to allow that. For now, you will have
> to
> > either use a new topic with more partitions or if you don't care about
> > messages being consumed in order, you can feed the consumed messages to a
> > separate thread pool for processing.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
> >
> > > Hi I have a few topics created with 1 partition. After running the
> > cluster
> > > for a few days, I want to increase the partition to 10 to improve the
> > > consumer throughput. I learnt that it is not supported in 0.8.
> > >
> > > What is the recommendation of improving consumer throughput after a
> topic
> > > is created and the data volume increased?
> > >
> > > Regards,
> > > Cal
> > >
> >
>
Re: Changing the number of partitions after a topic is created
Posted by Timothy Chen <tn...@gmail.com>.
Hi Jun,
I wonder when will the tool be available? We're very interested in changing
the number of partitions for a topic after creation too.
Thanks!
Tim
On Thu, Jul 4, 2013 at 9:06 PM, Jun Rao <ju...@gmail.com> wrote:
> Currently, once a topic is created, the number of partitions can't be
> changed. We are working on a tool to allow that. For now, you will have to
> either use a new topic with more partitions or if you don't care about
> messages being consumed in order, you can feed the consumed messages to a
> separate thread pool for processing.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
>
> > Hi I have a few topics created with 1 partition. After running the
> cluster
> > for a few days, I want to increase the partition to 10 to improve the
> > consumer throughput. I learnt that it is not supported in 0.8.
> >
> > What is the recommendation of improving consumer throughput after a topic
> > is created and the data volume increased?
> >
> > Regards,
> > Cal
> >
>
RE: Changing the number of partitions after a topic is created
Posted by "Yu, Libo " <li...@citi.com>.
Thanks, Jun. Have a good weekend.
Regards,
Libo
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, July 05, 2013 3:08 PM
To: users@kafka.apache.org
Subject: Re: Changing the number of partitions after a topic is created
No. However, you can specify the # of partitions when you create a topic using the create-topic command line tool.
Thanks,
Jun
On Fri, Jul 5, 2013 at 6:45 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Jun,
>
> Is it possible to specify a different partition number for each topic
> in the server.properties for 0.8?
> There is no documentation about that in the file.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, July 05, 2013 12:06 AM
> To: users@kafka.apache.org
> Subject: Re: Changing the number of partitions after a topic is
> created
>
> Currently, once a topic is created, the number of partitions can't be
> changed. We are working on a tool to allow that. For now, you will
> have to either use a new topic with more partitions or if you don't
> care about messages being consumed in order, you can feed the consumed
> messages to a separate thread pool for processing.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
>
> > Hi I have a few topics created with 1 partition. After running the
> > cluster for a few days, I want to increase the partition to 10 to
> > improve the consumer throughput. I learnt that it is not supported
> > in
> 0.8.
> >
> > What is the recommendation of improving consumer throughput after a
> > topic is created and the data volume increased?
> >
> > Regards,
> > Cal
> >
>
Re: Changing the number of partitions after a topic is created
Posted by Jun Rao <ju...@gmail.com>.
No. However, you can specify the # of partitions when you create a topic
using the create-topic command line tool.
Thanks,
Jun
On Fri, Jul 5, 2013 at 6:45 AM, Yu, Libo <li...@citi.com> wrote:
> Hi Jun,
>
> Is it possible to specify a different partition number for each topic in
> the server.properties for 0.8?
> There is no documentation about that in the file.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, July 05, 2013 12:06 AM
> To: users@kafka.apache.org
> Subject: Re: Changing the number of partitions after a topic is created
>
> Currently, once a topic is created, the number of partitions can't be
> changed. We are working on a tool to allow that. For now, you will have to
> either use a new topic with more partitions or if you don't care about
> messages being consumed in order, you can feed the consumed messages to a
> separate thread pool for processing.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
>
> > Hi I have a few topics created with 1 partition. After running the
> > cluster for a few days, I want to increase the partition to 10 to
> > improve the consumer throughput. I learnt that it is not supported in
> 0.8.
> >
> > What is the recommendation of improving consumer throughput after a
> > topic is created and the data volume increased?
> >
> > Regards,
> > Cal
> >
>
RE: Changing the number of partitions after a topic is created
Posted by "Yu, Libo " <li...@citi.com>.
Hi Jun,
Is it possible to specify a different partition number for each topic in the server.properties for 0.8?
There is no documentation about that in the file.
Regards,
Libo
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, July 05, 2013 12:06 AM
To: users@kafka.apache.org
Subject: Re: Changing the number of partitions after a topic is created
Currently, once a topic is created, the number of partitions can't be changed. We are working on a tool to allow that. For now, you will have to either use a new topic with more partitions or if you don't care about messages being consumed in order, you can feed the consumed messages to a separate thread pool for processing.
Thanks,
Jun
On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
> Hi I have a few topics created with 1 partition. After running the
> cluster for a few days, I want to increase the partition to 10 to
> improve the consumer throughput. I learnt that it is not supported in 0.8.
>
> What is the recommendation of improving consumer throughput after a
> topic is created and the data volume increased?
>
> Regards,
> Cal
>
Re: Changing the number of partitions after a topic is created
Posted by Jun Rao <ju...@gmail.com>.
Currently, once a topic is created, the number of partitions can't be
changed. We are working on a tool to allow that. For now, you will have to
either use a new topic with more partitions or if you don't care about
messages being consumed in order, you can feed the consumed messages to a
separate thread pool for processing.
Thanks,
Jun
On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <ck...@gmail.com> wrote:
> Hi I have a few topics created with 1 partition. After running the cluster
> for a few days, I want to increase the partition to 10 to improve the
> consumer throughput. I learnt that it is not supported in 0.8.
>
> What is the recommendation of improving consumer throughput after a topic
> is created and the data volume increased?
>
> Regards,
> Cal
>
Re: Changing the number of partitions after a topic is created
Posted by Philip O'Toole <ph...@loggly.com>.
On Thu, Jul 4, 2013 at 12:06 PM, Calvin Lei <ck...@gmail.com> wrote:
> Thanks Phillip. I used the kafka-topic.sh to create the topic to have 1 partition. Would the changing the server properties still work in this case?
I'm not familiar with that script, but server.properties is what
matters. Change that and it will work.
>
> Also this setting sounds like a global overwrite, correct?
Yes, but it can be set on a topic-by-topic basis if you prefer. Read
the comments contained within the server.properties for guidance.
https://github.com/apache/kafka/blob/0.7/config/server.properties
>
> regards,
> Cal
>
>
> On Jul 4, 2013, at 11:49 AM, Philip O'Toole <ph...@loggly.com> wrote:
>
>> If you can pause your Producers, simply change the partition count to
>> 10 in the Kafka server properties file, and restart it. If you use the
>> high-level consumer, it will automatically start draining all
>> partitions that exist for a given topic.
>>
>> And our Kafka producers always write to partition -1 for a given
>> topic, meaning they never care about the partition count, and need no
>> changes in circumstances like this.
>>
>> Philip
>>
>> On Thu, Jul 4, 2013 at 11:11 AM, Calvin Lei <ck...@gmail.com> wrote:
>>> Hi I have a few topics created with 1 partition. After running the cluster
>>> for a few days, I want to increase the partition to 10 to improve the
>>> consumer throughput. I learnt that it is not supported in 0.8.
>>>
>>> What is the recommendation of improving consumer throughput after a topic
>>> is created and the data volume increased?
>>>
>>> Regards,
>>> Cal
>
Re: Changing the number of partitions after a topic is created
Posted by Calvin Lei <ck...@gmail.com>.
Thanks Phillip. I used the kafka-topic.sh to create the topic to have 1 partition. Would the changing the server properties still work in this case?
Also this setting sounds like a global overwrite, correct?
regards,
Cal
On Jul 4, 2013, at 11:49 AM, Philip O'Toole <ph...@loggly.com> wrote:
> If you can pause your Producers, simply change the partition count to
> 10 in the Kafka server properties file, and restart it. If you use the
> high-level consumer, it will automatically start draining all
> partitions that exist for a given topic.
>
> And our Kafka producers always write to partition -1 for a given
> topic, meaning they never care about the partition count, and need no
> changes in circumstances like this.
>
> Philip
>
> On Thu, Jul 4, 2013 at 11:11 AM, Calvin Lei <ck...@gmail.com> wrote:
>> Hi I have a few topics created with 1 partition. After running the cluster
>> for a few days, I want to increase the partition to 10 to improve the
>> consumer throughput. I learnt that it is not supported in 0.8.
>>
>> What is the recommendation of improving consumer throughput after a topic
>> is created and the data volume increased?
>>
>> Regards,
>> Cal
Re: Changing the number of partitions after a topic is created
Posted by Philip O'Toole <ph...@loggly.com>.
If you can pause your Producers, simply change the partition count to
10 in the Kafka server properties file, and restart it. If you use the
high-level consumer, it will automatically start draining all
partitions that exist for a given topic.
And our Kafka producers always write to partition -1 for a given
topic, meaning they never care about the partition count, and need no
changes in circumstances like this.
Philip
On Thu, Jul 4, 2013 at 11:11 AM, Calvin Lei <ck...@gmail.com> wrote:
> Hi I have a few topics created with 1 partition. After running the cluster
> for a few days, I want to increase the partition to 10 to improve the
> consumer throughput. I learnt that it is not supported in 0.8.
>
> What is the recommendation of improving consumer throughput after a topic
> is created and the data volume increased?
>
> Regards,
> Cal