You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shone Sadler <sh...@gmail.com> on 2013/10/09 22:34:43 UTC
Add Partition and Rebalancing
Hello,
I am testing out the AddPartitionCommand in Kafka 8.0 and wanted to verify
the behavior I am seeing. Currently, running producers and consumers
appear unaffected by scaling out of partitions. i.e. Producers do not pick
up the new partition size for calls to the Partitioner and consumers are
not rebalanced to pick up the new partitions. Both producers and consumers
must be restarted to pick up the new partitions. Is this expected
behavior? I did come across the following ticket with respect to consumers
-> https://issues.apache.org/jira/browse/KAFKA-1075 which is labeled as
fixed, however I am not seeing the expected behavior.
Thanks,
Shone Sadler
Re: Add Partition and Rebalancing
Posted by Shone Sadler <sh...@gmail.com>.
Just verified the fix. Works great! Thanks for the quick response and fix
Guozhang.
Shone
On Wed, Oct 9, 2013 at 5:32 PM, Shone Sadler <sh...@gmail.com> wrote:
> Ahh yes, thanks Guozhang!
>
> I thought I did a pull yesterday, but the last change I had was for
> Kafka-1073 (below). I did a pull again and I now see the change. I'll run
> my tests again.
>
> Thanks,
> Shone
>
> shonesadler$ git log
> commit 71ed6ca3368ff38909f502565a4bf0f39e70fc6c
> Author: Jun Rao <ju...@gmail.com>
> Date: Mon Oct 7 09:22:12 2013 -0700
>
> kafka-1073; CheckReassignmentStatus is broken; patched by Jun Rao;
> reviewed by Guozhang Wang, Swa
>
>
>
> On Wed, Oct 9, 2013 at 5:04 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hello Shone,
>>
>> Could you do a git log to see if KAFKA-1075 has been included in your
>> repository? It was checked in this Monday, so if you checked out the code
>> earlier than that then the fix will not be included.
>>
>> Guozhang
>>
>>
>> On Wed, Oct 9, 2013 at 1:34 PM, Shone Sadler <sh...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> >
>> > I am testing out the AddPartitionCommand in Kafka 8.0 and wanted to
>> verify
>> > the behavior I am seeing. Currently, running producers and consumers
>> > appear unaffected by scaling out of partitions. i.e. Producers do not
>> pick
>> > up the new partition size for calls to the Partitioner and consumers are
>> > not rebalanced to pick up the new partitions. Both producers and
>> consumers
>> > must be restarted to pick up the new partitions. Is this expected
>> > behavior? I did come across the following ticket with respect to
>> consumers
>> > -> https://issues.apache.org/jira/browse/KAFKA-1075 which is labeled as
>> > fixed, however I am not seeing the expected behavior.
>> >
>> >
>> > Thanks,
>> > Shone Sadler
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
Re: Add Partition and Rebalancing
Posted by Shone Sadler <sh...@gmail.com>.
Ahh yes, thanks Guozhang!
I thought I did a pull yesterday, but the last change I had was for
Kafka-1073 (below). I did a pull again and I now see the change. I'll run
my tests again.
Thanks,
Shone
shonesadler$ git log
commit 71ed6ca3368ff38909f502565a4bf0f39e70fc6c
Author: Jun Rao <ju...@gmail.com>
Date: Mon Oct 7 09:22:12 2013 -0700
kafka-1073; CheckReassignmentStatus is broken; patched by Jun Rao;
reviewed by Guozhang Wang, Swa
On Wed, Oct 9, 2013 at 5:04 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Shone,
>
> Could you do a git log to see if KAFKA-1075 has been included in your
> repository? It was checked in this Monday, so if you checked out the code
> earlier than that then the fix will not be included.
>
> Guozhang
>
>
> On Wed, Oct 9, 2013 at 1:34 PM, Shone Sadler <sh...@gmail.com>
> wrote:
>
> > Hello,
> >
> >
> > I am testing out the AddPartitionCommand in Kafka 8.0 and wanted to
> verify
> > the behavior I am seeing. Currently, running producers and consumers
> > appear unaffected by scaling out of partitions. i.e. Producers do not
> pick
> > up the new partition size for calls to the Partitioner and consumers are
> > not rebalanced to pick up the new partitions. Both producers and
> consumers
> > must be restarted to pick up the new partitions. Is this expected
> > behavior? I did come across the following ticket with respect to
> consumers
> > -> https://issues.apache.org/jira/browse/KAFKA-1075 which is labeled as
> > fixed, however I am not seeing the expected behavior.
> >
> >
> > Thanks,
> > Shone Sadler
> >
>
>
>
> --
> -- Guozhang
>
Re: Add Partition and Rebalancing
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Shone,
Could you do a git log to see if KAFKA-1075 has been included in your
repository? It was checked in this Monday, so if you checked out the code
earlier than that then the fix will not be included.
Guozhang
On Wed, Oct 9, 2013 at 1:34 PM, Shone Sadler <sh...@gmail.com> wrote:
> Hello,
>
>
> I am testing out the AddPartitionCommand in Kafka 8.0 and wanted to verify
> the behavior I am seeing. Currently, running producers and consumers
> appear unaffected by scaling out of partitions. i.e. Producers do not pick
> up the new partition size for calls to the Partitioner and consumers are
> not rebalanced to pick up the new partitions. Both producers and consumers
> must be restarted to pick up the new partitions. Is this expected
> behavior? I did come across the following ticket with respect to consumers
> -> https://issues.apache.org/jira/browse/KAFKA-1075 which is labeled as
> fixed, however I am not seeing the expected behavior.
>
>
> Thanks,
> Shone Sadler
>
--
-- Guozhang