You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sebastian Schmitz <se...@propellerhead.co.nz> on 2019/09/26 02:28:37 UTC
One Partition missing a node in ISR
Hello,
I have one topic with 12 partitions where the partition 0 is missing one
node from ISR... Is there a way I get get it back to work again without
having to do some weird stuff like restarting the cluster?
Because this missing node in ISR is causing some problems for the
consumers...
Thx
Sebastian
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
I'll keep an eye on that and other topics in all my four environments to
see when it appears again.... For the moment it's fine
On 02-Oct-19 7:50 AM, M. Manna wrote:
> I don’t think you can control replication using clients (pub/sub). And
> neither should you be allowed to.
>
> I guess we need more of such examples to understand correct use if admin
> API.
>
> Thanks,
>
> On Tue, 1 Oct 2019 at 19:45, Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> I didn't change anything with the topic... But as it only affects one
>> topic my idea would be that it's related to the publisher as it's
>> different for this specific topic.
>>
>> Can a publisher tell Kafka what to do about replication? Like having
>> such an influence to have this kind of symptoms?
>>
>> Also the JMX-Metrics of Kafka didn't report any under-replicated
>> partitions... But when running the kafka-topics.sh with
>> --under-replicated-partitions it showed the ones from this topic.
>>
>>
>> On 01-Oct-19 10:58 PM, M. Manna wrote:
>>> I was going to ask you to do that :) As long as >1 replicas are in-sync
>>> Kafka handles this nicely for recreating everything in the restarted
>> broker.
>>> I am curious, do you remember manipulating something before all these
>>> started? e.g. Using some topic admin command (or something along the
>> line) ?
>>>
>>> On Tue, 1 Oct 2019 at 02:00, Sebastian Schmitz <
>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>
>>>> I deleted the topic now and with topic-auto-create enabled it was
>>>> immediately recreated and all is in sync again.
>>>>
>>>> Will keep and eye on this to see if it happens again....
>>>>
>>>>
>>>> On 30-Sep-19 3:12 PM, Sebastian Schmitz wrote:
>>>>> Hello again,
>>>>>
>>>>> after like 15 minutes I have now this result:
>>>>>
>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>>> --zookeeper node1:2181 --reassignment-json-file move2.json --verify
>>>>> Status of partition reassignment:
>>>>> Reassignment of partition my_topic-7 completed successfully
>>>>> Reassignment of partition my_topic-14 completed successfully
>>>>> Reassignment of partition my_topic-8 completed successfully
>>>>> Reassignment of partition my_topic-4 completed successfully
>>>>> Reassignment of partition my_topic-3 completed successfully
>>>>> Reassignment of partition my_topic-13 completed successfully
>>>>> Reassignment of partition my_topic-1 completed successfully
>>>>> Reassignment of partition my_topic-15 completed successfully
>>>>> Reassignment of partition my_topic-6 completed successfully
>>>>> Reassignment of partition my_topic-11 completed successfully
>>>>> Reassignment of partition my_topic-0 completed successfully
>>>>> Reassignment of partition my_topic-12 completed successfully
>>>>> Reassignment of partition my_topic-10 completed successfully
>>>>> Reassignment of partition my_topic-2 completed successfully
>>>>> Reassignment of partition my_topic-9 completed successfully
>>>>> Reassignment of partition my_topic-5 completed successfully
>>>>>
>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>>>> 2,3,1 Isr: 1
>>>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>>>> 3,1,2 Isr: 1
>>>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>>>> 1,2,3 Isr: 1
>>>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>>>> 2,1,3 Isr: 1
>>>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>>>> 3,2,1 Isr: 1
>>>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>>>> 1,3,2 Isr: 1
>>>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>>>> 2,3,1 Isr: 1
>>>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>>>> 3,1,2 Isr: 1
>>>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>>>> 1,2,3 Isr: 1
>>>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>>>> 2,1,3 Isr: 1
>>>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>>>> 3,2,1 Isr: 1
>>>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>>>> 1,3,2 Isr: 1
>>>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>>>> 2,3,1 Isr: 1,2,3
>>>>> Topic: my_topic Partition: 13 Leader: 3 Replicas:
>>>>> 3,1,2 Isr: 1,2,3
>>>>> Topic: my_topic Partition: 14 Leader: 1 Replicas:
>>>>> 1,2,3 Isr: 1,2,3
>>>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>>>> 2,1,3 Isr: 1,2,3
>>>>>
>>>>> I looks like it didn't help and also the reassignment caused some
>>>>> disconnects from Zookeeper on all nodes which triggered some alerts in
>>>>> my monitoring.
>>>>>
>>>>> I also checked the logs and found that those partitions had last
>>>>> activity on some days ago and last action was for each of them some
>>>>> rolling to new log segment. Which was logged on all three nodes and
>>>>> then it seems like only node1 remained ISR... It also didn't happen
>>>>> for all missing partitions at the same time. Partition 5 stopped on
>>>>> 27th, partition 8 stopped on 25th... And so far only one topic is
>>>>> affected.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>> On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I just ran the kafka-reassign-partitions with --generate to create
>>>>>> the json and then with --execute to run it.
>>>>>> Now when checking with --verify I can see that the 4 partitions (it
>>>>>> now changed from only one partitions not having all in ISR to 12 not
>>>>>> being all in ISR) are successful, but the others are still in
>>>>>> progress.... That status remains:
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>>>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>>>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>>>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1
>>>>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>>>>> 3,1,2 Isr: 1
>>>>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>>>>> 1,2,3 Isr: 1
>>>>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1
>>>>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>>>>> 3,2,1 Isr: 1
>>>>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>>>>> 1,3,2 Isr: 1
>>>>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1
>>>>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>>>>> 3,1,2 Isr: 1
>>>>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>>>>> 1,2,3 Isr: 1
>>>>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1
>>>>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>>>>> 3,2,1 Isr: 1
>>>>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>>>>> 1,3,2 Isr: 1
>>>>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>>>>>> 3,1,2 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>>>>>> 1,2,3 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1,3,2
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>>>> --reassignment-json-file move2.json --execute
>>>>>> Current partition replica assignment
>>>>>>
>>>>>>
>> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
>>>>>> Save this to use as the --reassignment-json-file option during
>> rollback
>>>>>> Successfully started reassignment of partitions.
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>>>> --reassignment-json-file move2.json --verify
>>>>>> Status of partition reassignment:
>>>>>> Reassignment of partition my_topic-7 is still in progress
>>>>>> Reassignment of partition my_topic-14 completed successfully
>>>>>> Reassignment of partition my_topic-8 is still in progress
>>>>>> Reassignment of partition my_topic-4 is still in progress
>>>>>> Reassignment of partition my_topic-3 is still in progress
>>>>>> Reassignment of partition my_topic-13 completed successfully
>>>>>> Reassignment of partition my_topic-1 is still in progress
>>>>>> Reassignment of partition my_topic-15 completed successfully
>>>>>> Reassignment of partition my_topic-6 is still in progress
>>>>>> Reassignment of partition my_topic-11 is still in progress
>>>>>> Reassignment of partition my_topic-0 is still in progress
>>>>>> Reassignment of partition my_topic-12 completed successfully
>>>>>> Reassignment of partition my_topic-10 is still in progress
>>>>>> Reassignment of partition my_topic-2 is still in progress
>>>>>> Reassignment of partition my_topic-9 is still in progress
>>>>>> Reassignment of partition my_topic-5 is still in progress
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>>>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>>>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>>>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1
>>>>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>>>>> 3,1,2 Isr: 1
>>>>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>>>>> 1,2,3 Isr: 1
>>>>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1
>>>>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>>>>> 3,2,1 Isr: 1
>>>>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>>>>> 1,3,2 Isr: 1
>>>>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1
>>>>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>>>>> 3,1,2 Isr: 1
>>>>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>>>>> 1,2,3 Isr: 1
>>>>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1
>>>>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>>>>> 3,2,1 Isr: 1
>>>>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>>>>> 1,3,2 Isr: 1
>>>>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>>>>> 2,3,1 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>>>>>> 3,1,2 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>>>>>> 1,2,3 Isr: 1,3,2
>>>>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>>>>> 2,1,3 Isr: 1,3,2
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>>>> --reassignment-json-file move2.json --verify
>>>>>> Status of partition reassignment:
>>>>>> Reassignment of partition my_topic-7 is still in progress
>>>>>> Reassignment of partition my_topic-14 completed successfully
>>>>>> Reassignment of partition my_topic-8 is still in progress
>>>>>> Reassignment of partition my_topic-4 is still in progress
>>>>>> Reassignment of partition my_topic-3 is still in progress
>>>>>> Reassignment of partition my_topic-13 completed successfully
>>>>>> Reassignment of partition my_topic-1 is still in progress
>>>>>> Reassignment of partition my_topic-15 completed successfully
>>>>>> Reassignment of partition my_topic-6 is still in progress
>>>>>> Reassignment of partition my_topic-11 is still in progress
>>>>>> Reassignment of partition my_topic-0 is still in progress
>>>>>> Reassignment of partition my_topic-12 completed successfully
>>>>>> Reassignment of partition my_topic-10 is still in progress
>>>>>> Reassignment of partition my_topic-2 is still in progress
>>>>>> Reassignment of partition my_topic-9 is still in progress
>>>>>> Reassignment of partition my_topic-5 is still in progress
>>>>>>
>>>>>> I also checked Zookeeper for active brokers:
>>>>>>
>>>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
>>>>>> node1:2181 ls /brokers/ids
>>>>>> Connecting to node1:2181
>>>>>>
>>>>>> WATCHER::
>>>>>>
>>>>>> WatchedEvent state:SyncConnected type:None path:null
>>>>>> [1, 2, 3]
>>>>>>
>>>>>> What's next?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>> On 26-Sep-19 10:04 PM, M. Manna wrote:
>>>>>>> hello,
>>>>>>>
>>>>>>> Could you please try to run kafka-reassign-partitions with your topic
>>>>>>> reassignment JSON? That doesn't require any restart, and should tell
>>>>>>> you if
>>>>>>> any issues with reassignment. The examples are provided in
>>>>>>> Confluence wiki.
>>>>>>>
>>>>>>> I would recommend that you do a "Describe" on your topic to ensure
>>>>>>> that all
>>>>>>> partitions and ISR metadata is up-to-date.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
>>>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have one topic with 12 partitions where the partition 0 is
>>>>>>>> missing one
>>>>>>>> node from ISR... Is there a way I get get it back to work again
>>>>>>>> without
>>>>>>>> having to do some weird stuff like restarting the cluster?
>>>>>>>> Because this missing node in ISR is causing some problems for the
>>>>>>>> consumers...
>>>>>>>>
>>>>>>>> Thx
>>>>>>>>
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> DISCLAIMER
>>>>>>>> This email contains information that is confidential and which
>>>>>>>> may be
>>>>>>>> legally privileged. If you have received this email in error please
>>>>>>>>
>>>>>>>> notify the sender immediately and delete the email.
>>>>>>>> This email is intended
>>>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>>>> disclose this email in any way.
>>>>>>>>
>>>> --
>>>> DISCLAIMER
>>>> This email contains information that is confidential and which
>>>> may be
>>>> legally privileged. If you have received this email in error please
>>>>
>>>> notify the sender immediately and delete the email.
>>>> This email is intended
>>>> solely for the use of the intended recipient and you may not use or
>>>> disclose this email in any way.
>>>>
>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by "M. Manna" <ma...@gmail.com>.
I don’t think you can control replication using clients (pub/sub). And
neither should you be allowed to.
I guess we need more of such examples to understand correct use if admin
API.
Thanks,
On Tue, 1 Oct 2019 at 19:45, Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> I didn't change anything with the topic... But as it only affects one
> topic my idea would be that it's related to the publisher as it's
> different for this specific topic.
>
> Can a publisher tell Kafka what to do about replication? Like having
> such an influence to have this kind of symptoms?
>
> Also the JMX-Metrics of Kafka didn't report any under-replicated
> partitions... But when running the kafka-topics.sh with
> --under-replicated-partitions it showed the ones from this topic.
>
>
> On 01-Oct-19 10:58 PM, M. Manna wrote:
> > I was going to ask you to do that :) As long as >1 replicas are in-sync
> > Kafka handles this nicely for recreating everything in the restarted
> broker.
> >
> > I am curious, do you remember manipulating something before all these
> > started? e.g. Using some topic admin command (or something along the
> line) ?
> >
> >
> > On Tue, 1 Oct 2019 at 02:00, Sebastian Schmitz <
> > sebastian.schmitz@propellerhead.co.nz> wrote:
> >
> >> I deleted the topic now and with topic-auto-create enabled it was
> >> immediately recreated and all is in sync again.
> >>
> >> Will keep and eye on this to see if it happens again....
> >>
> >>
> >> On 30-Sep-19 3:12 PM, Sebastian Schmitz wrote:
> >>> Hello again,
> >>>
> >>> after like 15 minutes I have now this result:
> >>>
> >>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >>> --zookeeper node1:2181 --reassignment-json-file move2.json --verify
> >>> Status of partition reassignment:
> >>> Reassignment of partition my_topic-7 completed successfully
> >>> Reassignment of partition my_topic-14 completed successfully
> >>> Reassignment of partition my_topic-8 completed successfully
> >>> Reassignment of partition my_topic-4 completed successfully
> >>> Reassignment of partition my_topic-3 completed successfully
> >>> Reassignment of partition my_topic-13 completed successfully
> >>> Reassignment of partition my_topic-1 completed successfully
> >>> Reassignment of partition my_topic-15 completed successfully
> >>> Reassignment of partition my_topic-6 completed successfully
> >>> Reassignment of partition my_topic-11 completed successfully
> >>> Reassignment of partition my_topic-0 completed successfully
> >>> Reassignment of partition my_topic-12 completed successfully
> >>> Reassignment of partition my_topic-10 completed successfully
> >>> Reassignment of partition my_topic-2 completed successfully
> >>> Reassignment of partition my_topic-9 completed successfully
> >>> Reassignment of partition my_topic-5 completed successfully
> >>>
> >>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> >>> --bootstrap-server localhost:9092 --topic my_topic --describe
> >>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >>>
> >>
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> >>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> >>> 2,3,1 Isr: 1
> >>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> >>> 3,1,2 Isr: 1
> >>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> >>> 1,2,3 Isr: 1
> >>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> >>> 2,1,3 Isr: 1
> >>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> >>> 3,2,1 Isr: 1
> >>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> >>> 1,3,2 Isr: 1
> >>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> >>> 2,3,1 Isr: 1
> >>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> >>> 3,1,2 Isr: 1
> >>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> >>> 1,2,3 Isr: 1
> >>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> >>> 2,1,3 Isr: 1
> >>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> >>> 3,2,1 Isr: 1
> >>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> >>> 1,3,2 Isr: 1
> >>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> >>> 2,3,1 Isr: 1,2,3
> >>> Topic: my_topic Partition: 13 Leader: 3 Replicas:
> >>> 3,1,2 Isr: 1,2,3
> >>> Topic: my_topic Partition: 14 Leader: 1 Replicas:
> >>> 1,2,3 Isr: 1,2,3
> >>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> >>> 2,1,3 Isr: 1,2,3
> >>>
> >>> I looks like it didn't help and also the reassignment caused some
> >>> disconnects from Zookeeper on all nodes which triggered some alerts in
> >>> my monitoring.
> >>>
> >>> I also checked the logs and found that those partitions had last
> >>> activity on some days ago and last action was for each of them some
> >>> rolling to new log segment. Which was logged on all three nodes and
> >>> then it seems like only node1 remained ISR... It also didn't happen
> >>> for all missing partitions at the same time. Partition 5 stopped on
> >>> 27th, partition 8 stopped on 25th... And so far only one topic is
> >>> affected.
> >>>
> >>> Thanks
> >>>
> >>> Sebastian
> >>>
> >>>
> >>> On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
> >>>> Hello,
> >>>>
> >>>> I just ran the kafka-reassign-partitions with --generate to create
> >>>> the json and then with --execute to run it.
> >>>> Now when checking with --verify I can see that the 4 partitions (it
> >>>> now changed from only one partitions not having all in ISR to 12 not
> >>>> being all in ISR) are successful, but the others are still in
> >>>> progress.... That status remains:
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> >>>> --bootstrap-server localhost:9092 --topic my_topic --describe
> >>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >>>>
> >>
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> >>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1
> >>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> >>>> 3,1,2 Isr: 1
> >>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> >>>> 1,2,3 Isr: 1
> >>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1
> >>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> >>>> 3,2,1 Isr: 1
> >>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> >>>> 1,3,2 Isr: 1
> >>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1
> >>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> >>>> 3,1,2 Isr: 1
> >>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> >>>> 1,2,3 Isr: 1
> >>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1
> >>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> >>>> 3,2,1 Isr: 1
> >>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> >>>> 1,3,2 Isr: 1
> >>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> >>>> 3,1,2 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> >>>> 1,2,3 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1,3,2
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >>>> --reassignment-json-file move2.json --execute
> >>>> Current partition replica assignment
> >>>>
> >>>>
> >>
> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
> >>
> >>>>
> >>>> Save this to use as the --reassignment-json-file option during
> rollback
> >>>> Successfully started reassignment of partitions.
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >>>> --reassignment-json-file move2.json --verify
> >>>> Status of partition reassignment:
> >>>> Reassignment of partition my_topic-7 is still in progress
> >>>> Reassignment of partition my_topic-14 completed successfully
> >>>> Reassignment of partition my_topic-8 is still in progress
> >>>> Reassignment of partition my_topic-4 is still in progress
> >>>> Reassignment of partition my_topic-3 is still in progress
> >>>> Reassignment of partition my_topic-13 completed successfully
> >>>> Reassignment of partition my_topic-1 is still in progress
> >>>> Reassignment of partition my_topic-15 completed successfully
> >>>> Reassignment of partition my_topic-6 is still in progress
> >>>> Reassignment of partition my_topic-11 is still in progress
> >>>> Reassignment of partition my_topic-0 is still in progress
> >>>> Reassignment of partition my_topic-12 completed successfully
> >>>> Reassignment of partition my_topic-10 is still in progress
> >>>> Reassignment of partition my_topic-2 is still in progress
> >>>> Reassignment of partition my_topic-9 is still in progress
> >>>> Reassignment of partition my_topic-5 is still in progress
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> >>>> --bootstrap-server localhost:9092 --topic my_topic --describe
> >>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >>>>
> >>
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> >>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1
> >>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> >>>> 3,1,2 Isr: 1
> >>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> >>>> 1,2,3 Isr: 1
> >>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1
> >>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> >>>> 3,2,1 Isr: 1
> >>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> >>>> 1,3,2 Isr: 1
> >>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1
> >>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> >>>> 3,1,2 Isr: 1
> >>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> >>>> 1,2,3 Isr: 1
> >>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1
> >>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> >>>> 3,2,1 Isr: 1
> >>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> >>>> 1,3,2 Isr: 1
> >>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> >>>> 2,3,1 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> >>>> 3,1,2 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> >>>> 1,2,3 Isr: 1,3,2
> >>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> >>>> 2,1,3 Isr: 1,3,2
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >>>> --reassignment-json-file move2.json --verify
> >>>> Status of partition reassignment:
> >>>> Reassignment of partition my_topic-7 is still in progress
> >>>> Reassignment of partition my_topic-14 completed successfully
> >>>> Reassignment of partition my_topic-8 is still in progress
> >>>> Reassignment of partition my_topic-4 is still in progress
> >>>> Reassignment of partition my_topic-3 is still in progress
> >>>> Reassignment of partition my_topic-13 completed successfully
> >>>> Reassignment of partition my_topic-1 is still in progress
> >>>> Reassignment of partition my_topic-15 completed successfully
> >>>> Reassignment of partition my_topic-6 is still in progress
> >>>> Reassignment of partition my_topic-11 is still in progress
> >>>> Reassignment of partition my_topic-0 is still in progress
> >>>> Reassignment of partition my_topic-12 completed successfully
> >>>> Reassignment of partition my_topic-10 is still in progress
> >>>> Reassignment of partition my_topic-2 is still in progress
> >>>> Reassignment of partition my_topic-9 is still in progress
> >>>> Reassignment of partition my_topic-5 is still in progress
> >>>>
> >>>> I also checked Zookeeper for active brokers:
> >>>>
> >>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
> >>>> node1:2181 ls /brokers/ids
> >>>> Connecting to node1:2181
> >>>>
> >>>> WATCHER::
> >>>>
> >>>> WatchedEvent state:SyncConnected type:None path:null
> >>>> [1, 2, 3]
> >>>>
> >>>> What's next?
> >>>>
> >>>> Thanks
> >>>>
> >>>> Sebastian
> >>>>
> >>>>
> >>>> On 26-Sep-19 10:04 PM, M. Manna wrote:
> >>>>> hello,
> >>>>>
> >>>>> Could you please try to run kafka-reassign-partitions with your topic
> >>>>> reassignment JSON? That doesn't require any restart, and should tell
> >>>>> you if
> >>>>> any issues with reassignment. The examples are provided in
> >>>>> Confluence wiki.
> >>>>>
> >>>>> I would recommend that you do a "Describe" on your topic to ensure
> >>>>> that all
> >>>>> partitions and ISR metadata is up-to-date.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
> >>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I have one topic with 12 partitions where the partition 0 is
> >>>>>> missing one
> >>>>>> node from ISR... Is there a way I get get it back to work again
> >>>>>> without
> >>>>>> having to do some weird stuff like restarting the cluster?
> >>>>>> Because this missing node in ISR is causing some problems for the
> >>>>>> consumers...
> >>>>>>
> >>>>>> Thx
> >>>>>>
> >>>>>> Sebastian
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> DISCLAIMER
> >>>>>> This email contains information that is confidential and which
> >>>>>> may be
> >>>>>> legally privileged. If you have received this email in error please
> >>>>>>
> >>>>>> notify the sender immediately and delete the email.
> >>>>>> This email is intended
> >>>>>> solely for the use of the intended recipient and you may not use or
> >>>>>> disclose this email in any way.
> >>>>>>
> >> --
> >> DISCLAIMER
> >> This email contains information that is confidential and which
> >> may be
> >> legally privileged. If you have received this email in error please
> >>
> >> notify the sender immediately and delete the email.
> >> This email is intended
> >> solely for the use of the intended recipient and you may not use or
> >> disclose this email in any way.
> >>
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: One Partition missing a node in ISR
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
I didn't change anything with the topic... But as it only affects one
topic my idea would be that it's related to the publisher as it's
different for this specific topic.
Can a publisher tell Kafka what to do about replication? Like having
such an influence to have this kind of symptoms?
Also the JMX-Metrics of Kafka didn't report any under-replicated
partitions... But when running the kafka-topics.sh with
--under-replicated-partitions it showed the ones from this topic.
On 01-Oct-19 10:58 PM, M. Manna wrote:
> I was going to ask you to do that :) As long as >1 replicas are in-sync
> Kafka handles this nicely for recreating everything in the restarted broker.
>
> I am curious, do you remember manipulating something before all these
> started? e.g. Using some topic admin command (or something along the line) ?
>
>
> On Tue, 1 Oct 2019 at 02:00, Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> I deleted the topic now and with topic-auto-create enabled it was
>> immediately recreated and all is in sync again.
>>
>> Will keep and eye on this to see if it happens again....
>>
>>
>> On 30-Sep-19 3:12 PM, Sebastian Schmitz wrote:
>>> Hello again,
>>>
>>> after like 15 minutes I have now this result:
>>>
>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>> --zookeeper node1:2181 --reassignment-json-file move2.json --verify
>>> Status of partition reassignment:
>>> Reassignment of partition my_topic-7 completed successfully
>>> Reassignment of partition my_topic-14 completed successfully
>>> Reassignment of partition my_topic-8 completed successfully
>>> Reassignment of partition my_topic-4 completed successfully
>>> Reassignment of partition my_topic-3 completed successfully
>>> Reassignment of partition my_topic-13 completed successfully
>>> Reassignment of partition my_topic-1 completed successfully
>>> Reassignment of partition my_topic-15 completed successfully
>>> Reassignment of partition my_topic-6 completed successfully
>>> Reassignment of partition my_topic-11 completed successfully
>>> Reassignment of partition my_topic-0 completed successfully
>>> Reassignment of partition my_topic-12 completed successfully
>>> Reassignment of partition my_topic-10 completed successfully
>>> Reassignment of partition my_topic-2 completed successfully
>>> Reassignment of partition my_topic-9 completed successfully
>>> Reassignment of partition my_topic-5 completed successfully
>>>
>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>> 2,3,1 Isr: 1
>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>> 3,1,2 Isr: 1
>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>> 1,2,3 Isr: 1
>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>> 2,1,3 Isr: 1
>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>> 3,2,1 Isr: 1
>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>> 1,3,2 Isr: 1
>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>> 2,3,1 Isr: 1
>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>> 3,1,2 Isr: 1
>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>> 1,2,3 Isr: 1
>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>> 2,1,3 Isr: 1
>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>> 3,2,1 Isr: 1
>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>> 1,3,2 Isr: 1
>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>> 2,3,1 Isr: 1,2,3
>>> Topic: my_topic Partition: 13 Leader: 3 Replicas:
>>> 3,1,2 Isr: 1,2,3
>>> Topic: my_topic Partition: 14 Leader: 1 Replicas:
>>> 1,2,3 Isr: 1,2,3
>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>> 2,1,3 Isr: 1,2,3
>>>
>>> I looks like it didn't help and also the reassignment caused some
>>> disconnects from Zookeeper on all nodes which triggered some alerts in
>>> my monitoring.
>>>
>>> I also checked the logs and found that those partitions had last
>>> activity on some days ago and last action was for each of them some
>>> rolling to new log segment. Which was logged on all three nodes and
>>> then it seems like only node1 remained ISR... It also didn't happen
>>> for all missing partitions at the same time. Partition 5 stopped on
>>> 27th, partition 8 stopped on 25th... And so far only one topic is
>>> affected.
>>>
>>> Thanks
>>>
>>> Sebastian
>>>
>>>
>>> On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
>>>> Hello,
>>>>
>>>> I just ran the kafka-reassign-partitions with --generate to create
>>>> the json and then with --execute to run it.
>>>> Now when checking with --verify I can see that the 4 partitions (it
>>>> now changed from only one partitions not having all in ISR to 12 not
>>>> being all in ISR) are successful, but the others are still in
>>>> progress.... That status remains:
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1
>>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>>> 3,1,2 Isr: 1
>>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>>> 1,2,3 Isr: 1
>>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1
>>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>>> 3,2,1 Isr: 1
>>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>>> 1,3,2 Isr: 1
>>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1
>>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>>> 3,1,2 Isr: 1
>>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>>> 1,2,3 Isr: 1
>>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1
>>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>>> 3,2,1 Isr: 1
>>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>>> 1,3,2 Isr: 1
>>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1,3,2
>>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>>>> 3,1,2 Isr: 1,3,2
>>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>>>> 1,2,3 Isr: 1,3,2
>>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1,3,2
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>> --reassignment-json-file move2.json --execute
>>>> Current partition replica assignment
>>>>
>>>>
>> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
>>
>>>>
>>>> Save this to use as the --reassignment-json-file option during rollback
>>>> Successfully started reassignment of partitions.
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>> --reassignment-json-file move2.json --verify
>>>> Status of partition reassignment:
>>>> Reassignment of partition my_topic-7 is still in progress
>>>> Reassignment of partition my_topic-14 completed successfully
>>>> Reassignment of partition my_topic-8 is still in progress
>>>> Reassignment of partition my_topic-4 is still in progress
>>>> Reassignment of partition my_topic-3 is still in progress
>>>> Reassignment of partition my_topic-13 completed successfully
>>>> Reassignment of partition my_topic-1 is still in progress
>>>> Reassignment of partition my_topic-15 completed successfully
>>>> Reassignment of partition my_topic-6 is still in progress
>>>> Reassignment of partition my_topic-11 is still in progress
>>>> Reassignment of partition my_topic-0 is still in progress
>>>> Reassignment of partition my_topic-12 completed successfully
>>>> Reassignment of partition my_topic-10 is still in progress
>>>> Reassignment of partition my_topic-2 is still in progress
>>>> Reassignment of partition my_topic-9 is still in progress
>>>> Reassignment of partition my_topic-5 is still in progress
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>>>> --bootstrap-server localhost:9092 --topic my_topic --describe
>>>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>>>>
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>>>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1
>>>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>>>> 3,1,2 Isr: 1
>>>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>>>> 1,2,3 Isr: 1
>>>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1
>>>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>>>> 3,2,1 Isr: 1
>>>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>>>> 1,3,2 Isr: 1
>>>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1
>>>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>>>> 3,1,2 Isr: 1
>>>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>>>> 1,2,3 Isr: 1
>>>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1
>>>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>>>> 3,2,1 Isr: 1
>>>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>>>> 1,3,2 Isr: 1
>>>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>>>> 2,3,1 Isr: 1,3,2
>>>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>>>> 3,1,2 Isr: 1,3,2
>>>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>>>> 1,2,3 Isr: 1,3,2
>>>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>>>> 2,1,3 Isr: 1,3,2
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>>>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>>>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>>>> --reassignment-json-file move2.json --verify
>>>> Status of partition reassignment:
>>>> Reassignment of partition my_topic-7 is still in progress
>>>> Reassignment of partition my_topic-14 completed successfully
>>>> Reassignment of partition my_topic-8 is still in progress
>>>> Reassignment of partition my_topic-4 is still in progress
>>>> Reassignment of partition my_topic-3 is still in progress
>>>> Reassignment of partition my_topic-13 completed successfully
>>>> Reassignment of partition my_topic-1 is still in progress
>>>> Reassignment of partition my_topic-15 completed successfully
>>>> Reassignment of partition my_topic-6 is still in progress
>>>> Reassignment of partition my_topic-11 is still in progress
>>>> Reassignment of partition my_topic-0 is still in progress
>>>> Reassignment of partition my_topic-12 completed successfully
>>>> Reassignment of partition my_topic-10 is still in progress
>>>> Reassignment of partition my_topic-2 is still in progress
>>>> Reassignment of partition my_topic-9 is still in progress
>>>> Reassignment of partition my_topic-5 is still in progress
>>>>
>>>> I also checked Zookeeper for active brokers:
>>>>
>>>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
>>>> node1:2181 ls /brokers/ids
>>>> Connecting to node1:2181
>>>>
>>>> WATCHER::
>>>>
>>>> WatchedEvent state:SyncConnected type:None path:null
>>>> [1, 2, 3]
>>>>
>>>> What's next?
>>>>
>>>> Thanks
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> On 26-Sep-19 10:04 PM, M. Manna wrote:
>>>>> hello,
>>>>>
>>>>> Could you please try to run kafka-reassign-partitions with your topic
>>>>> reassignment JSON? That doesn't require any restart, and should tell
>>>>> you if
>>>>> any issues with reassignment. The examples are provided in
>>>>> Confluence wiki.
>>>>>
>>>>> I would recommend that you do a "Describe" on your topic to ensure
>>>>> that all
>>>>> partitions and ISR metadata is up-to-date.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
>>>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have one topic with 12 partitions where the partition 0 is
>>>>>> missing one
>>>>>> node from ISR... Is there a way I get get it back to work again
>>>>>> without
>>>>>> having to do some weird stuff like restarting the cluster?
>>>>>> Because this missing node in ISR is causing some problems for the
>>>>>> consumers...
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>>
>>>>>> --
>>>>>> DISCLAIMER
>>>>>> This email contains information that is confidential and which
>>>>>> may be
>>>>>> legally privileged. If you have received this email in error please
>>>>>>
>>>>>> notify the sender immediately and delete the email.
>>>>>> This email is intended
>>>>>> solely for the use of the intended recipient and you may not use or
>>>>>> disclose this email in any way.
>>>>>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by "M. Manna" <ma...@gmail.com>.
I was going to ask you to do that :) As long as >1 replicas are in-sync
Kafka handles this nicely for recreating everything in the restarted broker.
I am curious, do you remember manipulating something before all these
started? e.g. Using some topic admin command (or something along the line) ?
On Tue, 1 Oct 2019 at 02:00, Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> I deleted the topic now and with topic-auto-create enabled it was
> immediately recreated and all is in sync again.
>
> Will keep and eye on this to see if it happens again....
>
>
> On 30-Sep-19 3:12 PM, Sebastian Schmitz wrote:
> > Hello again,
> >
> > after like 15 minutes I have now this result:
> >
> > root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> > ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> > --zookeeper node1:2181 --reassignment-json-file move2.json --verify
> > Status of partition reassignment:
> > Reassignment of partition my_topic-7 completed successfully
> > Reassignment of partition my_topic-14 completed successfully
> > Reassignment of partition my_topic-8 completed successfully
> > Reassignment of partition my_topic-4 completed successfully
> > Reassignment of partition my_topic-3 completed successfully
> > Reassignment of partition my_topic-13 completed successfully
> > Reassignment of partition my_topic-1 completed successfully
> > Reassignment of partition my_topic-15 completed successfully
> > Reassignment of partition my_topic-6 completed successfully
> > Reassignment of partition my_topic-11 completed successfully
> > Reassignment of partition my_topic-0 completed successfully
> > Reassignment of partition my_topic-12 completed successfully
> > Reassignment of partition my_topic-10 completed successfully
> > Reassignment of partition my_topic-2 completed successfully
> > Reassignment of partition my_topic-9 completed successfully
> > Reassignment of partition my_topic-5 completed successfully
> >
> > root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> > --bootstrap-server localhost:9092 --topic my_topic --describe
> > Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> > Topic: my_topic Partition: 0 Leader: 1 Replicas:
> > 2,3,1 Isr: 1
> > Topic: my_topic Partition: 1 Leader: 1 Replicas:
> > 3,1,2 Isr: 1
> > Topic: my_topic Partition: 2 Leader: 1 Replicas:
> > 1,2,3 Isr: 1
> > Topic: my_topic Partition: 3 Leader: 1 Replicas:
> > 2,1,3 Isr: 1
> > Topic: my_topic Partition: 4 Leader: 1 Replicas:
> > 3,2,1 Isr: 1
> > Topic: my_topic Partition: 5 Leader: 1 Replicas:
> > 1,3,2 Isr: 1
> > Topic: my_topic Partition: 6 Leader: 1 Replicas:
> > 2,3,1 Isr: 1
> > Topic: my_topic Partition: 7 Leader: 1 Replicas:
> > 3,1,2 Isr: 1
> > Topic: my_topic Partition: 8 Leader: 1 Replicas:
> > 1,2,3 Isr: 1
> > Topic: my_topic Partition: 9 Leader: 1 Replicas:
> > 2,1,3 Isr: 1
> > Topic: my_topic Partition: 10 Leader: 1 Replicas:
> > 3,2,1 Isr: 1
> > Topic: my_topic Partition: 11 Leader: 1 Replicas:
> > 1,3,2 Isr: 1
> > Topic: my_topic Partition: 12 Leader: 1 Replicas:
> > 2,3,1 Isr: 1,2,3
> > Topic: my_topic Partition: 13 Leader: 3 Replicas:
> > 3,1,2 Isr: 1,2,3
> > Topic: my_topic Partition: 14 Leader: 1 Replicas:
> > 1,2,3 Isr: 1,2,3
> > Topic: my_topic Partition: 15 Leader: 1 Replicas:
> > 2,1,3 Isr: 1,2,3
> >
> > I looks like it didn't help and also the reassignment caused some
> > disconnects from Zookeeper on all nodes which triggered some alerts in
> > my monitoring.
> >
> > I also checked the logs and found that those partitions had last
> > activity on some days ago and last action was for each of them some
> > rolling to new log segment. Which was logged on all three nodes and
> > then it seems like only node1 remained ISR... It also didn't happen
> > for all missing partitions at the same time. Partition 5 stopped on
> > 27th, partition 8 stopped on 25th... And so far only one topic is
> > affected.
> >
> > Thanks
> >
> > Sebastian
> >
> >
> > On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
> >> Hello,
> >>
> >> I just ran the kafka-reassign-partitions with --generate to create
> >> the json and then with --execute to run it.
> >> Now when checking with --verify I can see that the 4 partitions (it
> >> now changed from only one partitions not having all in ISR to 12 not
> >> being all in ISR) are successful, but the others are still in
> >> progress.... That status remains:
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> >> --bootstrap-server localhost:9092 --topic my_topic --describe
> >> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >>
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> >> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1
> >> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> >> 3,1,2 Isr: 1
> >> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> >> 1,2,3 Isr: 1
> >> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1
> >> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> >> 3,2,1 Isr: 1
> >> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> >> 1,3,2 Isr: 1
> >> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1
> >> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> >> 3,1,2 Isr: 1
> >> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> >> 1,2,3 Isr: 1
> >> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1
> >> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> >> 3,2,1 Isr: 1
> >> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> >> 1,3,2 Isr: 1
> >> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1,3,2
> >> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> >> 3,1,2 Isr: 1,3,2
> >> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> >> 1,2,3 Isr: 1,3,2
> >> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1,3,2
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >> --reassignment-json-file move2.json --execute
> >> Current partition replica assignment
> >>
> >>
> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
>
> >>
> >>
> >> Save this to use as the --reassignment-json-file option during rollback
> >> Successfully started reassignment of partitions.
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >> --reassignment-json-file move2.json --verify
> >> Status of partition reassignment:
> >> Reassignment of partition my_topic-7 is still in progress
> >> Reassignment of partition my_topic-14 completed successfully
> >> Reassignment of partition my_topic-8 is still in progress
> >> Reassignment of partition my_topic-4 is still in progress
> >> Reassignment of partition my_topic-3 is still in progress
> >> Reassignment of partition my_topic-13 completed successfully
> >> Reassignment of partition my_topic-1 is still in progress
> >> Reassignment of partition my_topic-15 completed successfully
> >> Reassignment of partition my_topic-6 is still in progress
> >> Reassignment of partition my_topic-11 is still in progress
> >> Reassignment of partition my_topic-0 is still in progress
> >> Reassignment of partition my_topic-12 completed successfully
> >> Reassignment of partition my_topic-10 is still in progress
> >> Reassignment of partition my_topic-2 is still in progress
> >> Reassignment of partition my_topic-9 is still in progress
> >> Reassignment of partition my_topic-5 is still in progress
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> >> --bootstrap-server localhost:9092 --topic my_topic --describe
> >> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> >>
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> >> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1
> >> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> >> 3,1,2 Isr: 1
> >> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> >> 1,2,3 Isr: 1
> >> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1
> >> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> >> 3,2,1 Isr: 1
> >> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> >> 1,3,2 Isr: 1
> >> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1
> >> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> >> 3,1,2 Isr: 1
> >> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> >> 1,2,3 Isr: 1
> >> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1
> >> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> >> 3,2,1 Isr: 1
> >> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> >> 1,3,2 Isr: 1
> >> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> >> 2,3,1 Isr: 1,3,2
> >> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> >> 3,1,2 Isr: 1,3,2
> >> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> >> 1,2,3 Isr: 1,3,2
> >> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> >> 2,1,3 Isr: 1,3,2
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> >> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> >> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> >> --reassignment-json-file move2.json --verify
> >> Status of partition reassignment:
> >> Reassignment of partition my_topic-7 is still in progress
> >> Reassignment of partition my_topic-14 completed successfully
> >> Reassignment of partition my_topic-8 is still in progress
> >> Reassignment of partition my_topic-4 is still in progress
> >> Reassignment of partition my_topic-3 is still in progress
> >> Reassignment of partition my_topic-13 completed successfully
> >> Reassignment of partition my_topic-1 is still in progress
> >> Reassignment of partition my_topic-15 completed successfully
> >> Reassignment of partition my_topic-6 is still in progress
> >> Reassignment of partition my_topic-11 is still in progress
> >> Reassignment of partition my_topic-0 is still in progress
> >> Reassignment of partition my_topic-12 completed successfully
> >> Reassignment of partition my_topic-10 is still in progress
> >> Reassignment of partition my_topic-2 is still in progress
> >> Reassignment of partition my_topic-9 is still in progress
> >> Reassignment of partition my_topic-5 is still in progress
> >>
> >> I also checked Zookeeper for active brokers:
> >>
> >> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
> >> node1:2181 ls /brokers/ids
> >> Connecting to node1:2181
> >>
> >> WATCHER::
> >>
> >> WatchedEvent state:SyncConnected type:None path:null
> >> [1, 2, 3]
> >>
> >> What's next?
> >>
> >> Thanks
> >>
> >> Sebastian
> >>
> >>
> >> On 26-Sep-19 10:04 PM, M. Manna wrote:
> >>> hello,
> >>>
> >>> Could you please try to run kafka-reassign-partitions with your topic
> >>> reassignment JSON? That doesn't require any restart, and should tell
> >>> you if
> >>> any issues with reassignment. The examples are provided in
> >>> Confluence wiki.
> >>>
> >>> I would recommend that you do a "Describe" on your topic to ensure
> >>> that all
> >>> partitions and ISR metadata is up-to-date.
> >>>
> >>> Thanks,
> >>>
> >>>
> >>>
> >>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
> >>> sebastian.schmitz@propellerhead.co.nz> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> I have one topic with 12 partitions where the partition 0 is
> >>>> missing one
> >>>> node from ISR... Is there a way I get get it back to work again
> >>>> without
> >>>> having to do some weird stuff like restarting the cluster?
> >>>> Because this missing node in ISR is causing some problems for the
> >>>> consumers...
> >>>>
> >>>> Thx
> >>>>
> >>>> Sebastian
> >>>>
> >>>>
> >>>> --
> >>>> DISCLAIMER
> >>>> This email contains information that is confidential and which
> >>>> may be
> >>>> legally privileged. If you have received this email in error please
> >>>>
> >>>> notify the sender immediately and delete the email.
> >>>> This email is intended
> >>>> solely for the use of the intended recipient and you may not use or
> >>>> disclose this email in any way.
> >>>>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>
Re: One Partition missing a node in ISR
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
I deleted the topic now and with topic-auto-create enabled it was
immediately recreated and all is in sync again.
Will keep and eye on this to see if it happens again....
On 30-Sep-19 3:12 PM, Sebastian Schmitz wrote:
> Hello again,
>
> after like 15 minutes I have now this result:
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> --zookeeper node1:2181 --reassignment-json-file move2.json --verify
> Status of partition reassignment:
> Reassignment of partition my_topic-7 completed successfully
> Reassignment of partition my_topic-14 completed successfully
> Reassignment of partition my_topic-8 completed successfully
> Reassignment of partition my_topic-4 completed successfully
> Reassignment of partition my_topic-3 completed successfully
> Reassignment of partition my_topic-13 completed successfully
> Reassignment of partition my_topic-1 completed successfully
> Reassignment of partition my_topic-15 completed successfully
> Reassignment of partition my_topic-6 completed successfully
> Reassignment of partition my_topic-11 completed successfully
> Reassignment of partition my_topic-0 completed successfully
> Reassignment of partition my_topic-12 completed successfully
> Reassignment of partition my_topic-10 completed successfully
> Reassignment of partition my_topic-2 completed successfully
> Reassignment of partition my_topic-9 completed successfully
> Reassignment of partition my_topic-5 completed successfully
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> --bootstrap-server localhost:9092 --topic my_topic --describe
> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> 2,3,1 Isr: 1,2,3
> Topic: my_topic Partition: 13 Leader: 3 Replicas:
> 3,1,2 Isr: 1,2,3
> Topic: my_topic Partition: 14 Leader: 1 Replicas:
> 1,2,3 Isr: 1,2,3
> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> 2,1,3 Isr: 1,2,3
>
> I looks like it didn't help and also the reassignment caused some
> disconnects from Zookeeper on all nodes which triggered some alerts in
> my monitoring.
>
> I also checked the logs and found that those partitions had last
> activity on some days ago and last action was for each of them some
> rolling to new log segment. Which was logged on all three nodes and
> then it seems like only node1 remained ISR... It also didn't happen
> for all missing partitions at the same time. Partition 5 stopped on
> 27th, partition 8 stopped on 25th... And so far only one topic is
> affected.
>
> Thanks
>
> Sebastian
>
>
> On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
>> Hello,
>>
>> I just ran the kafka-reassign-partitions with --generate to create
>> the json and then with --execute to run it.
>> Now when checking with --verify I can see that the 4 partitions (it
>> now changed from only one partitions not having all in ISR to 12 not
>> being all in ISR) are successful, but the others are still in
>> progress.... That status remains:
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>> --bootstrap-server localhost:9092 --topic my_topic --describe
>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>> 2,3,1 Isr: 1
>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>> 3,1,2 Isr: 1
>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>> 1,2,3 Isr: 1
>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>> 2,1,3 Isr: 1
>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>> 3,2,1 Isr: 1
>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>> 1,3,2 Isr: 1
>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>> 2,3,1 Isr: 1
>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>> 3,1,2 Isr: 1
>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>> 1,2,3 Isr: 1
>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>> 2,1,3 Isr: 1
>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>> 3,2,1 Isr: 1
>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>> 1,3,2 Isr: 1
>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>> 2,3,1 Isr: 1,3,2
>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>> 3,1,2 Isr: 1,3,2
>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>> 1,2,3 Isr: 1,3,2
>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>> 2,1,3 Isr: 1,3,2
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>> --reassignment-json-file move2.json --execute
>> Current partition replica assignment
>>
>> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
>>
>>
>> Save this to use as the --reassignment-json-file option during rollback
>> Successfully started reassignment of partitions.
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>> --reassignment-json-file move2.json --verify
>> Status of partition reassignment:
>> Reassignment of partition my_topic-7 is still in progress
>> Reassignment of partition my_topic-14 completed successfully
>> Reassignment of partition my_topic-8 is still in progress
>> Reassignment of partition my_topic-4 is still in progress
>> Reassignment of partition my_topic-3 is still in progress
>> Reassignment of partition my_topic-13 completed successfully
>> Reassignment of partition my_topic-1 is still in progress
>> Reassignment of partition my_topic-15 completed successfully
>> Reassignment of partition my_topic-6 is still in progress
>> Reassignment of partition my_topic-11 is still in progress
>> Reassignment of partition my_topic-0 is still in progress
>> Reassignment of partition my_topic-12 completed successfully
>> Reassignment of partition my_topic-10 is still in progress
>> Reassignment of partition my_topic-2 is still in progress
>> Reassignment of partition my_topic-9 is still in progress
>> Reassignment of partition my_topic-5 is still in progress
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
>> --bootstrap-server localhost:9092 --topic my_topic --describe
>> Topic:my_topic PartitionCount:16 ReplicationFactor:3
>> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
>> Topic: my_topic Partition: 0 Leader: 1 Replicas:
>> 2,3,1 Isr: 1
>> Topic: my_topic Partition: 1 Leader: 1 Replicas:
>> 3,1,2 Isr: 1
>> Topic: my_topic Partition: 2 Leader: 1 Replicas:
>> 1,2,3 Isr: 1
>> Topic: my_topic Partition: 3 Leader: 1 Replicas:
>> 2,1,3 Isr: 1
>> Topic: my_topic Partition: 4 Leader: 1 Replicas:
>> 3,2,1 Isr: 1
>> Topic: my_topic Partition: 5 Leader: 1 Replicas:
>> 1,3,2 Isr: 1
>> Topic: my_topic Partition: 6 Leader: 1 Replicas:
>> 2,3,1 Isr: 1
>> Topic: my_topic Partition: 7 Leader: 1 Replicas:
>> 3,1,2 Isr: 1
>> Topic: my_topic Partition: 8 Leader: 1 Replicas:
>> 1,2,3 Isr: 1
>> Topic: my_topic Partition: 9 Leader: 1 Replicas:
>> 2,1,3 Isr: 1
>> Topic: my_topic Partition: 10 Leader: 1 Replicas:
>> 3,2,1 Isr: 1
>> Topic: my_topic Partition: 11 Leader: 1 Replicas:
>> 1,3,2 Isr: 1
>> Topic: my_topic Partition: 12 Leader: 1 Replicas:
>> 2,3,1 Isr: 1,3,2
>> Topic: my_topic Partition: 13 Leader: 2 Replicas:
>> 3,1,2 Isr: 1,3,2
>> Topic: my_topic Partition: 14 Leader: 3 Replicas:
>> 1,2,3 Isr: 1,3,2
>> Topic: my_topic Partition: 15 Leader: 1 Replicas:
>> 2,1,3 Isr: 1,3,2
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
>> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
>> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
>> --reassignment-json-file move2.json --verify
>> Status of partition reassignment:
>> Reassignment of partition my_topic-7 is still in progress
>> Reassignment of partition my_topic-14 completed successfully
>> Reassignment of partition my_topic-8 is still in progress
>> Reassignment of partition my_topic-4 is still in progress
>> Reassignment of partition my_topic-3 is still in progress
>> Reassignment of partition my_topic-13 completed successfully
>> Reassignment of partition my_topic-1 is still in progress
>> Reassignment of partition my_topic-15 completed successfully
>> Reassignment of partition my_topic-6 is still in progress
>> Reassignment of partition my_topic-11 is still in progress
>> Reassignment of partition my_topic-0 is still in progress
>> Reassignment of partition my_topic-12 completed successfully
>> Reassignment of partition my_topic-10 is still in progress
>> Reassignment of partition my_topic-2 is still in progress
>> Reassignment of partition my_topic-9 is still in progress
>> Reassignment of partition my_topic-5 is still in progress
>>
>> I also checked Zookeeper for active brokers:
>>
>> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
>> node1:2181 ls /brokers/ids
>> Connecting to node1:2181
>>
>> WATCHER::
>>
>> WatchedEvent state:SyncConnected type:None path:null
>> [1, 2, 3]
>>
>> What's next?
>>
>> Thanks
>>
>> Sebastian
>>
>>
>> On 26-Sep-19 10:04 PM, M. Manna wrote:
>>> hello,
>>>
>>> Could you please try to run kafka-reassign-partitions with your topic
>>> reassignment JSON? That doesn't require any restart, and should tell
>>> you if
>>> any issues with reassignment. The examples are provided in
>>> Confluence wiki.
>>>
>>> I would recommend that you do a "Describe" on your topic to ensure
>>> that all
>>> partitions and ISR metadata is up-to-date.
>>>
>>> Thanks,
>>>
>>>
>>>
>>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
>>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have one topic with 12 partitions where the partition 0 is
>>>> missing one
>>>> node from ISR... Is there a way I get get it back to work again
>>>> without
>>>> having to do some weird stuff like restarting the cluster?
>>>> Because this missing node in ISR is causing some problems for the
>>>> consumers...
>>>>
>>>> Thx
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> --
>>>> DISCLAIMER
>>>> This email contains information that is confidential and which
>>>> may be
>>>> legally privileged. If you have received this email in error please
>>>>
>>>> notify the sender immediately and delete the email.
>>>> This email is intended
>>>> solely for the use of the intended recipient and you may not use or
>>>> disclose this email in any way.
>>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello again,
after like 15 minutes I have now this result:
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
--zookeeper node1:2181 --reassignment-json-file move2.json --verify
Status of partition reassignment:
Reassignment of partition my_topic-7 completed successfully
Reassignment of partition my_topic-14 completed successfully
Reassignment of partition my_topic-8 completed successfully
Reassignment of partition my_topic-4 completed successfully
Reassignment of partition my_topic-3 completed successfully
Reassignment of partition my_topic-13 completed successfully
Reassignment of partition my_topic-1 completed successfully
Reassignment of partition my_topic-15 completed successfully
Reassignment of partition my_topic-6 completed successfully
Reassignment of partition my_topic-11 completed successfully
Reassignment of partition my_topic-0 completed successfully
Reassignment of partition my_topic-12 completed successfully
Reassignment of partition my_topic-10 completed successfully
Reassignment of partition my_topic-2 completed successfully
Reassignment of partition my_topic-9 completed successfully
Reassignment of partition my_topic-5 completed successfully
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
--bootstrap-server localhost:9092 --topic my_topic --describe
Topic:my_topic PartitionCount:16 ReplicationFactor:3
Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
Topic: my_topic Partition: 0 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 1 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 2 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 3 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 4 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 5 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 6 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 7 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 8 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 9 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 10 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 11 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 12 Leader: 1 Replicas: 2,3,1
Isr: 1,2,3
Topic: my_topic Partition: 13 Leader: 3 Replicas: 3,1,2
Isr: 1,2,3
Topic: my_topic Partition: 14 Leader: 1 Replicas: 1,2,3
Isr: 1,2,3
Topic: my_topic Partition: 15 Leader: 1 Replicas: 2,1,3
Isr: 1,2,3
I looks like it didn't help and also the reassignment caused some
disconnects from Zookeeper on all nodes which triggered some alerts in
my monitoring.
I also checked the logs and found that those partitions had last
activity on some days ago and last action was for each of them some
rolling to new log segment. Which was logged on all three nodes and then
it seems like only node1 remained ISR... It also didn't happen for all
missing partitions at the same time. Partition 5 stopped on 27th,
partition 8 stopped on 25th... And so far only one topic is affected.
Thanks
Sebastian
On 30-Sep-19 2:56 PM, Sebastian Schmitz wrote:
> Hello,
>
> I just ran the kafka-reassign-partitions with --generate to create the
> json and then with --execute to run it.
> Now when checking with --verify I can see that the 4 partitions (it
> now changed from only one partitions not having all in ISR to 12 not
> being all in ISR) are successful, but the others are still in
> progress.... That status remains:
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> --bootstrap-server localhost:9092 --topic my_topic --describe
> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> 2,3,1 Isr: 1,3,2
> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> 3,1,2 Isr: 1,3,2
> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> 1,2,3 Isr: 1,3,2
> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> 2,1,3 Isr: 1,3,2
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> --reassignment-json-file move2.json --execute
> Current partition replica assignment
>
> {"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
>
>
> Save this to use as the --reassignment-json-file option during rollback
> Successfully started reassignment of partitions.
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> --reassignment-json-file move2.json --verify
> Status of partition reassignment:
> Reassignment of partition my_topic-7 is still in progress
> Reassignment of partition my_topic-14 completed successfully
> Reassignment of partition my_topic-8 is still in progress
> Reassignment of partition my_topic-4 is still in progress
> Reassignment of partition my_topic-3 is still in progress
> Reassignment of partition my_topic-13 completed successfully
> Reassignment of partition my_topic-1 is still in progress
> Reassignment of partition my_topic-15 completed successfully
> Reassignment of partition my_topic-6 is still in progress
> Reassignment of partition my_topic-11 is still in progress
> Reassignment of partition my_topic-0 is still in progress
> Reassignment of partition my_topic-12 completed successfully
> Reassignment of partition my_topic-10 is still in progress
> Reassignment of partition my_topic-2 is still in progress
> Reassignment of partition my_topic-9 is still in progress
> Reassignment of partition my_topic-5 is still in progress
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
> --bootstrap-server localhost:9092 --topic my_topic --describe
> Topic:my_topic PartitionCount:16 ReplicationFactor:3
> Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
> Topic: my_topic Partition: 0 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 1 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 2 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 3 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 4 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 5 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 6 Leader: 1 Replicas:
> 2,3,1 Isr: 1
> Topic: my_topic Partition: 7 Leader: 1 Replicas:
> 3,1,2 Isr: 1
> Topic: my_topic Partition: 8 Leader: 1 Replicas:
> 1,2,3 Isr: 1
> Topic: my_topic Partition: 9 Leader: 1 Replicas:
> 2,1,3 Isr: 1
> Topic: my_topic Partition: 10 Leader: 1 Replicas:
> 3,2,1 Isr: 1
> Topic: my_topic Partition: 11 Leader: 1 Replicas:
> 1,3,2 Isr: 1
> Topic: my_topic Partition: 12 Leader: 1 Replicas:
> 2,3,1 Isr: 1,3,2
> Topic: my_topic Partition: 13 Leader: 2 Replicas:
> 3,1,2 Isr: 1,3,2
> Topic: my_topic Partition: 14 Leader: 3 Replicas:
> 1,2,3 Isr: 1,3,2
> Topic: my_topic Partition: 15 Leader: 1 Replicas:
> 2,1,3 Isr: 1,3,2
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
> ./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
> --zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
> --reassignment-json-file move2.json --verify
> Status of partition reassignment:
> Reassignment of partition my_topic-7 is still in progress
> Reassignment of partition my_topic-14 completed successfully
> Reassignment of partition my_topic-8 is still in progress
> Reassignment of partition my_topic-4 is still in progress
> Reassignment of partition my_topic-3 is still in progress
> Reassignment of partition my_topic-13 completed successfully
> Reassignment of partition my_topic-1 is still in progress
> Reassignment of partition my_topic-15 completed successfully
> Reassignment of partition my_topic-6 is still in progress
> Reassignment of partition my_topic-11 is still in progress
> Reassignment of partition my_topic-0 is still in progress
> Reassignment of partition my_topic-12 completed successfully
> Reassignment of partition my_topic-10 is still in progress
> Reassignment of partition my_topic-2 is still in progress
> Reassignment of partition my_topic-9 is still in progress
> Reassignment of partition my_topic-5 is still in progress
>
> I also checked Zookeeper for active brokers:
>
> root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
> node1:2181 ls /brokers/ids
> Connecting to node1:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [1, 2, 3]
>
> What's next?
>
> Thanks
>
> Sebastian
>
>
> On 26-Sep-19 10:04 PM, M. Manna wrote:
>> hello,
>>
>> Could you please try to run kafka-reassign-partitions with your topic
>> reassignment JSON? That doesn't require any restart, and should tell
>> you if
>> any issues with reassignment. The examples are provided in Confluence
>> wiki.
>>
>> I would recommend that you do a "Describe" on your topic to ensure
>> that all
>> partitions and ISR metadata is up-to-date.
>>
>> Thanks,
>>
>>
>>
>> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
>> sebastian.schmitz@propellerhead.co.nz> wrote:
>>
>>> Hello,
>>>
>>> I have one topic with 12 partitions where the partition 0 is missing
>>> one
>>> node from ISR... Is there a way I get get it back to work again without
>>> having to do some weird stuff like restarting the cluster?
>>> Because this missing node in ISR is causing some problems for the
>>> consumers...
>>>
>>> Thx
>>>
>>> Sebastian
>>>
>>>
>>> --
>>> DISCLAIMER
>>> This email contains information that is confidential and which
>>> may be
>>> legally privileged. If you have received this email in error please
>>>
>>> notify the sender immediately and delete the email.
>>> This email is intended
>>> solely for the use of the intended recipient and you may not use or
>>> disclose this email in any way.
>>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by Sebastian Schmitz <se...@propellerhead.co.nz>.
Hello,
I just ran the kafka-reassign-partitions with --generate to create the
json and then with --execute to run it.
Now when checking with --verify I can see that the 4 partitions (it now
changed from only one partitions not having all in ISR to 12 not being
all in ISR) are successful, but the others are still in progress....
That status remains:
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
--bootstrap-server localhost:9092 --topic my_topic --describe
Topic:my_topic PartitionCount:16 ReplicationFactor:3
Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
Topic: my_topic Partition: 0 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 1 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 2 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 3 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 4 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 5 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 6 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 7 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 8 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 9 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 10 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 11 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 12 Leader: 1 Replicas: 2,3,1
Isr: 1,3,2
Topic: my_topic Partition: 13 Leader: 2 Replicas: 3,1,2
Isr: 1,3,2
Topic: my_topic Partition: 14 Leader: 3 Replicas: 1,2,3
Isr: 1,3,2
Topic: my_topic Partition: 15 Leader: 1 Replicas: 2,1,3
Isr: 1,3,2
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
--zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
--reassignment-json-file move2.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
--zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
--reassignment-json-file move2.json --verify
Status of partition reassignment:
Reassignment of partition my_topic-7 is still in progress
Reassignment of partition my_topic-14 completed successfully
Reassignment of partition my_topic-8 is still in progress
Reassignment of partition my_topic-4 is still in progress
Reassignment of partition my_topic-3 is still in progress
Reassignment of partition my_topic-13 completed successfully
Reassignment of partition my_topic-1 is still in progress
Reassignment of partition my_topic-15 completed successfully
Reassignment of partition my_topic-6 is still in progress
Reassignment of partition my_topic-11 is still in progress
Reassignment of partition my_topic-0 is still in progress
Reassignment of partition my_topic-12 completed successfully
Reassignment of partition my_topic-10 is still in progress
Reassignment of partition my_topic-2 is still in progress
Reassignment of partition my_topic-9 is still in progress
Reassignment of partition my_topic-5 is still in progress
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh
--bootstrap-server localhost:9092 --topic my_topic --describe
Topic:my_topic PartitionCount:16 ReplicationFactor:3
Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824
Topic: my_topic Partition: 0 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 1 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 2 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 3 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 4 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 5 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 6 Leader: 1 Replicas: 2,3,1
Isr: 1
Topic: my_topic Partition: 7 Leader: 1 Replicas: 3,1,2
Isr: 1
Topic: my_topic Partition: 8 Leader: 1 Replicas: 1,2,3
Isr: 1
Topic: my_topic Partition: 9 Leader: 1 Replicas: 2,1,3
Isr: 1
Topic: my_topic Partition: 10 Leader: 1 Replicas: 3,2,1
Isr: 1
Topic: my_topic Partition: 11 Leader: 1 Replicas: 1,3,2
Isr: 1
Topic: my_topic Partition: 12 Leader: 1 Replicas: 2,3,1
Isr: 1,3,2
Topic: my_topic Partition: 13 Leader: 2 Replicas: 3,1,2
Isr: 1,3,2
Topic: my_topic Partition: 14 Leader: 3 Replicas: 1,2,3
Isr: 1,3,2
Topic: my_topic Partition: 15 Leader: 1 Replicas: 2,1,3
Isr: 1,3,2
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#
./kafka-reassign-partitions.sh --bootstrap-server localhost:9092
--zookeeper atazkafkp01.aucklandtransport.govt.nz:2181
--reassignment-json-file move2.json --verify
Status of partition reassignment:
Reassignment of partition my_topic-7 is still in progress
Reassignment of partition my_topic-14 completed successfully
Reassignment of partition my_topic-8 is still in progress
Reassignment of partition my_topic-4 is still in progress
Reassignment of partition my_topic-3 is still in progress
Reassignment of partition my_topic-13 completed successfully
Reassignment of partition my_topic-1 is still in progress
Reassignment of partition my_topic-15 completed successfully
Reassignment of partition my_topic-6 is still in progress
Reassignment of partition my_topic-11 is still in progress
Reassignment of partition my_topic-0 is still in progress
Reassignment of partition my_topic-12 completed successfully
Reassignment of partition my_topic-10 is still in progress
Reassignment of partition my_topic-2 is still in progress
Reassignment of partition my_topic-9 is still in progress
Reassignment of partition my_topic-5 is still in progress
I also checked Zookeeper for active brokers:
root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.sh
node1:2181 ls /brokers/ids
Connecting to node1:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[1, 2, 3]
What's next?
Thanks
Sebastian
On 26-Sep-19 10:04 PM, M. Manna wrote:
> hello,
>
> Could you please try to run kafka-reassign-partitions with your topic
> reassignment JSON? That doesn't require any restart, and should tell you if
> any issues with reassignment. The examples are provided in Confluence wiki.
>
> I would recommend that you do a "Describe" on your topic to ensure that all
> partitions and ISR metadata is up-to-date.
>
> Thanks,
>
>
>
> On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
> sebastian.schmitz@propellerhead.co.nz> wrote:
>
>> Hello,
>>
>> I have one topic with 12 partitions where the partition 0 is missing one
>> node from ISR... Is there a way I get get it back to work again without
>> having to do some weird stuff like restarting the cluster?
>> Because this missing node in ISR is causing some problems for the
>> consumers...
>>
>> Thx
>>
>> Sebastian
>>
>>
>> --
>> DISCLAIMER
>> This email contains information that is confidential and which
>> may be
>> legally privileged. If you have received this email in error please
>>
>> notify the sender immediately and delete the email.
>> This email is intended
>> solely for the use of the intended recipient and you may not use or
>> disclose this email in any way.
>>
--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please
notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.
Re: One Partition missing a node in ISR
Posted by "M. Manna" <ma...@gmail.com>.
hello,
Could you please try to run kafka-reassign-partitions with your topic
reassignment JSON? That doesn't require any restart, and should tell you if
any issues with reassignment. The examples are provided in Confluence wiki.
I would recommend that you do a "Describe" on your topic to ensure that all
partitions and ISR metadata is up-to-date.
Thanks,
On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
sebastian.schmitz@propellerhead.co.nz> wrote:
> Hello,
>
> I have one topic with 12 partitions where the partition 0 is missing one
> node from ISR... Is there a way I get get it back to work again without
> having to do some weird stuff like restarting the cluster?
> Because this missing node in ISR is causing some problems for the
> consumers...
>
> Thx
>
> Sebastian
>
>
> --
> DISCLAIMER
> This email contains information that is confidential and which
> may be
> legally privileged. If you have received this email in error please
>
> notify the sender immediately and delete the email.
> This email is intended
> solely for the use of the intended recipient and you may not use or
> disclose this email in any way.
>