You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID> on 2014/06/24 00:24:10 UTC

Kafka 'reassign-partitions' behavior if the replica does not catches up

I attempted bin/kafka-reassign-partitions.sh for the topics in my cluster. It was automatic reassignment of topics to new brokers.

Kafka duly initiated the partition reassignment. I believe kafka controller will do the new leader election once the re-assigned partitions are in sync.

However, in my case I realized that the new re-assigned partitions are not getting in sync at all. The in-rate in the existing partition is far more then the sync rate between earlier and newly reassigned partitions.

In this scenario, will kafka wait indefinitely for the newly assigned partition to become sync (which will never be).

Secondly what are the options to speed up the partition replica sync rate?

Running kafka 0.8.0

Regards,
Virendra

Re: Kafka 'reassign-partitions' behavior if the replica does not catches up

Posted by Neha Narkhede <ne...@gmail.com>.
We have a JIRA to track the cancel feature -
https://issues.apache.org/jira/browse/KAFKA-1506.

Thanks,
Neha


On Tue, Jun 24, 2014 at 1:32 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:

> In process of giving 0.8.1.1 a try.
>
> However I believe the question still holds true.
> If the amount of data getting pumped in a partition is such that any new
> replica partition can never catch up, then what would the partition
> reassignment tool behavior?
> If it will be in an infinitely wait mode waiting for the replica to be in
> sync, then is there an alternative to cancel the reassignment operation?
>
> Regards,
> Virendra
> On 6/23/14, 4:49 PM, "Neha Narkhede" <ne...@gmail.com> wrote:
>
> >However, in my case I realized that the new re-assigned partitions are not
> >getting in sync at all. The in-rate in the existing partition is far more
> >then the sync rate between earlier and newly reassigned partitions.
> >
> >This is probably due to a bug. Partition reassignment is stable in
> >0.8.1.1.
> >I recommend you upgrade and retry.
> >
> >Thanks
> >Neha
> >
> >
> >On Mon, Jun 23, 2014 at 3:24 PM, Virendra Pratap Singh <
> >vpsingh@yahoo-inc.com.invalid> wrote:
> >
> >> I attempted bin/kafka-reassign-partitions.sh for the topics in my
> >>cluster.
> >> It was automatic reassignment of topics to new brokers.
> >>
> >> Kafka duly initiated the partition reassignment. I believe kafka
> >> controller will do the new leader election once the re-assigned
> >>partitions
> >> are in sync.
> >>
> >> However, in my case I realized that the new re-assigned partitions are
> >>not
> >> getting in sync at all. The in-rate in the existing partition is far
> >>more
> >> then the sync rate between earlier and newly reassigned partitions.
> >>
> >> In this scenario, will kafka wait indefinitely for the newly assigned
> >> partition to become sync (which will never be).
> >>
> >> Secondly what are the options to speed up the partition replica sync
> >>rate?
> >>
> >> Running kafka 0.8.0
> >>
> >> Regards,
> >> Virendra
> >>
> >>
> >>
> >>
>
>

Re: Kafka 'reassign-partitions' behavior if the replica does not catches up

Posted by Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>.
In process of giving 0.8.1.1 a try.

However I believe the question still holds true.
If the amount of data getting pumped in a partition is such that any new
replica partition can never catch up, then what would the partition
reassignment tool behavior?
If it will be in an infinitely wait mode waiting for the replica to be in
sync, then is there an alternative to cancel the reassignment operation?

Regards,
Virendra
On 6/23/14, 4:49 PM, "Neha Narkhede" <ne...@gmail.com> wrote:

>However, in my case I realized that the new re-assigned partitions are not
>getting in sync at all. The in-rate in the existing partition is far more
>then the sync rate between earlier and newly reassigned partitions.
>
>This is probably due to a bug. Partition reassignment is stable in
>0.8.1.1.
>I recommend you upgrade and retry.
>
>Thanks
>Neha
>
>
>On Mon, Jun 23, 2014 at 3:24 PM, Virendra Pratap Singh <
>vpsingh@yahoo-inc.com.invalid> wrote:
>
>> I attempted bin/kafka-reassign-partitions.sh for the topics in my
>>cluster.
>> It was automatic reassignment of topics to new brokers.
>>
>> Kafka duly initiated the partition reassignment. I believe kafka
>> controller will do the new leader election once the re-assigned
>>partitions
>> are in sync.
>>
>> However, in my case I realized that the new re-assigned partitions are
>>not
>> getting in sync at all. The in-rate in the existing partition is far
>>more
>> then the sync rate between earlier and newly reassigned partitions.
>>
>> In this scenario, will kafka wait indefinitely for the newly assigned
>> partition to become sync (which will never be).
>>
>> Secondly what are the options to speed up the partition replica sync
>>rate?
>>
>> Running kafka 0.8.0
>>
>> Regards,
>> Virendra
>>
>>
>>
>>

Re: Kafka 'reassign-partitions' behavior if the replica does not catches up

Posted by Neha Narkhede <ne...@gmail.com>.
However, in my case I realized that the new re-assigned partitions are not
getting in sync at all. The in-rate in the existing partition is far more
then the sync rate between earlier and newly reassigned partitions.

This is probably due to a bug. Partition reassignment is stable in 0.8.1.1.
I recommend you upgrade and retry.

Thanks
Neha


On Mon, Jun 23, 2014 at 3:24 PM, Virendra Pratap Singh <
vpsingh@yahoo-inc.com.invalid> wrote:

> I attempted bin/kafka-reassign-partitions.sh for the topics in my cluster.
> It was automatic reassignment of topics to new brokers.
>
> Kafka duly initiated the partition reassignment. I believe kafka
> controller will do the new leader election once the re-assigned partitions
> are in sync.
>
> However, in my case I realized that the new re-assigned partitions are not
> getting in sync at all. The in-rate in the existing partition is far more
> then the sync rate between earlier and newly reassigned partitions.
>
> In this scenario, will kafka wait indefinitely for the newly assigned
> partition to become sync (which will never be).
>
> Secondly what are the options to speed up the partition replica sync rate?
>
> Running kafka 0.8.0
>
> Regards,
> Virendra
>
>
>
>