You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Thomas KIEFFER <th...@olamobile.com.INVALID> on 2017/03/13 09:43:18 UTC
Lost ISR when upgrading kafka from 0.10.0.1 to any newer version like
0.10.1.0 or 0.10.2.0
I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw
that ISR is lost when I upgrade one instance. I didn't find out yet
anything relevant about this problem, logs seems just fine.
eg.
kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
Topic:redirects PartitionCount:6 ReplicationFactor:2
Configs:retention.bytes=10737418240
Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2
Isr: 1,2
Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0
Isr: 2
Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1
Isr: 1
Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0
Isr: 1
Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1
Isr: 2,1
Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2
Isr: 2
It run with Zookeeper 3.4.6.
As those clusters are in production, I didn't try to migrate more than 1
instance after spotting this ISR problem, and then rollback to the
original version 0.10.0.1.
Any update about this would be greatly receive.
--
<https://fr.linkedin.com/in/thomas-kieffer-28517324>
Thomas Kieffer
Senior Linux Systems Administrator
Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 |
www.olamobile.com <http://www.olamobile.com>
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version
like 0.10.1.0 or 0.10.2.0
Posted by Ismael Juma <is...@juma.me.uk>.
Looking at the output you pasted, broker `0` was the one being upgraded? A
few things to check:
1. Does broker `0` connect to the other brokers after the restart
2. Is broker `0` able to connect to zookeeper
3. Does everything look OK in the controller and state-change logs in the
controller node
4. Did you allow enough time for the restarted broker to rejoin the ISR
Ismael
On Tue, Mar 14, 2017 at 1:37 PM, Thomas KIEFFER <
thomas.kieffer@olamobile.com.invalid> wrote:
> Yes, I've set the inter.broker.protocol.version=0.10.0 before restarting
> each broker on a previous update. Clusters currently run with this config.
>
> On 03/14/2017 12:34 PM, Ismael Juma wrote:
>
> So, to double-check, you set inter.broker.protocol.version=0.10.0 before
> bouncing each broker?
>
> On Tue, Mar 14, 2017 at 11:22 AM, Thomas KIEFFER <th...@olamobile.com.invalid> wrote:
>
>
> Hello Ismael,
>
> Thank you for your feedback.
>
> Yes I've done this changes on a previous upgrade and set them accordingly
> with the new version when trying to do the upgrade.
>
> inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0,
> 0.10.0 or 0.10.1).
> log.message.format.version=CURRENT_KAFKA_VERSION (See potential
> performance impact following the upgrade for the details on what this
> configuration does.)
> On 03/14/2017 11:26 AM, Ismael Juma wrote:
>
> Hi Thomas,
>
> Did you follow the instructions:https://kafka.apache.org/documentation/#upgrade
>
> Ismael
>
> On Mon, Mar 13, 2017 at 9:43 AM, Thomas KIEFFER <th...@olamobile.com.invalid> <th...@olamobile.com.invalid> wrote:
>
>
> I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
> I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw that
> ISR is lost when I upgrade one instance. I didn't find out yet anything
> relevant about this problem, logs seems just fine.
> eg.
>
> kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
> Topic:redirects PartitionCount:6 ReplicationFactor:2
> Configs:retention.bytes=10737418240
> Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2 Isr:
> 1,2
> Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0 Isr:
> 2
> Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1 Isr:
> 1
> Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0 Isr:
> 1
> Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1 Isr:
> 2,1
> Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2 Isr:
> 2
>
> It run with Zookeeper 3.4.6.
>
> As those clusters are in production, I didn't try to migrate more than 1
> instance after spotting this ISR problem, and then rollback to the original
> version 0.10.0.1.
>
> Any update about this would be greatly receive.
>
> --
> <https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 <+352%20691%20444%20263> <+352%20691%20444%20263>
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
>
> --<https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 <+352%20691%20444%20263>
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
>
>
> --
> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version
like 0.10.1.0 or 0.10.2.0
Posted by Thomas KIEFFER <th...@olamobile.com.INVALID>.
Yes, I've set the inter.broker.protocol.version=0.10.0 before restarting
each broker on a previous update. Clusters currently run with this config.
On 03/14/2017 12:34 PM, Ismael Juma wrote:
> So, to double-check, you set inter.broker.protocol.version=0.10.0 before
> bouncing each broker?
>
> On Tue, Mar 14, 2017 at 11:22 AM, Thomas KIEFFER <
> thomas.kieffer@olamobile.com.invalid> wrote:
>
>> Hello Ismael,
>>
>> Thank you for your feedback.
>>
>> Yes I've done this changes on a previous upgrade and set them accordingly
>> with the new version when trying to do the upgrade.
>>
>> inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0,
>> 0.10.0 or 0.10.1).
>> log.message.format.version=CURRENT_KAFKA_VERSION (See potential
>> performance impact following the upgrade for the details on what this
>> configuration does.)
>> On 03/14/2017 11:26 AM, Ismael Juma wrote:
>>
>> Hi Thomas,
>>
>> Did you follow the instructions:
>> https://kafka.apache.org/documentation/#upgrade
>>
>> Ismael
>>
>> On Mon, Mar 13, 2017 at 9:43 AM, Thomas KIEFFER <th...@olamobile.com.invalid> wrote:
>>
>>
>> I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
>> I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw that
>> ISR is lost when I upgrade one instance. I didn't find out yet anything
>> relevant about this problem, logs seems just fine.
>> eg.
>>
>> kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
>> Topic:redirects PartitionCount:6 ReplicationFactor:2
>> Configs:retention.bytes=10737418240
>> Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2 Isr:
>> 1,2
>> Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0 Isr:
>> 2
>> Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1 Isr:
>> 1
>> Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0 Isr:
>> 1
>> Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1 Isr:
>> 2,1
>> Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2 Isr:
>> 2
>>
>> It run with Zookeeper 3.4.6.
>>
>> As those clusters are in production, I didn't try to migrate more than 1
>> instance after spotting this ISR problem, and then rollback to the original
>> version 0.10.0.1.
>>
>> Any update about this would be greatly receive.
>>
>> --
>> <https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>>
>> Thomas Kieffer
>>
>> Senior Linux Systems Administrator
>>
>> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 <+352%20691%20444%20263>
>> <+352%20691%20444%20263> | www.olamobile.com
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission, dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipient is prohibited. If you received
>> this in error, please contact the sender and delete the material from any
>> computer.
>>
>>
>> --
>> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>>
>> Thomas Kieffer
>>
>> Senior Linux Systems Administrator
>>
>> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263
>> <+352%20691%20444%20263> | www.olamobile.com
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission, dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipient is prohibited. If you received
>> this in error, please contact the sender and delete the material from any
>> computer.
>>
--
<https://fr.linkedin.com/in/thomas-kieffer-28517324>
Thomas Kieffer
Senior Linux Systems Administrator
Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 |
www.olamobile.com <http://www.olamobile.com>
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version
like 0.10.1.0 or 0.10.2.0
Posted by Ismael Juma <is...@juma.me.uk>.
So, to double-check, you set inter.broker.protocol.version=0.10.0 before
bouncing each broker?
On Tue, Mar 14, 2017 at 11:22 AM, Thomas KIEFFER <
thomas.kieffer@olamobile.com.invalid> wrote:
> Hello Ismael,
>
> Thank you for your feedback.
>
> Yes I've done this changes on a previous upgrade and set them accordingly
> with the new version when trying to do the upgrade.
>
> inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0,
> 0.10.0 or 0.10.1).
> log.message.format.version=CURRENT_KAFKA_VERSION (See potential
> performance impact following the upgrade for the details on what this
> configuration does.)
> On 03/14/2017 11:26 AM, Ismael Juma wrote:
>
> Hi Thomas,
>
> Did you follow the instructions:
> https://kafka.apache.org/documentation/#upgrade
>
> Ismael
>
> On Mon, Mar 13, 2017 at 9:43 AM, Thomas KIEFFER <th...@olamobile.com.invalid> wrote:
>
>
> I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
> I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw that
> ISR is lost when I upgrade one instance. I didn't find out yet anything
> relevant about this problem, logs seems just fine.
> eg.
>
> kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
> Topic:redirects PartitionCount:6 ReplicationFactor:2
> Configs:retention.bytes=10737418240
> Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2 Isr:
> 1,2
> Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0 Isr:
> 2
> Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1 Isr:
> 1
> Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0 Isr:
> 1
> Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1 Isr:
> 2,1
> Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2 Isr:
> 2
>
> It run with Zookeeper 3.4.6.
>
> As those clusters are in production, I didn't try to migrate more than 1
> instance after spotting this ISR problem, and then rollback to the original
> version 0.10.0.1.
>
> Any update about this would be greatly receive.
>
> --
> <https://fr.linkedin.com/in/thomas-kieffer-28517324> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 <+352%20691%20444%20263>
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
>
> --
> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version
like 0.10.1.0 or 0.10.2.0
Posted by Thomas KIEFFER <th...@olamobile.com.INVALID>.
Hello Ismael,
Thank you for your feedback.
Yes I've done this changes on a previous upgrade and set them
accordingly with the new version when trying to do the upgrade.
inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2, 0.9.0,
0.10.0 or 0.10.1).
log.message.format.version=CURRENT_KAFKA_VERSION (See potential
performance impact following the upgrade for the details on what this
configuration does.)
On 03/14/2017 11:26 AM, Ismael Juma wrote:
> Hi Thomas,
>
> Did you follow the instructions:
>
> https://kafka.apache.org/documentation/#upgrade
>
> Ismael
>
> On Mon, Mar 13, 2017 at 9:43 AM, Thomas KIEFFER <
> thomas.kieffer@olamobile.com.invalid> wrote:
>
>> I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
>> I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw that
>> ISR is lost when I upgrade one instance. I didn't find out yet anything
>> relevant about this problem, logs seems just fine.
>> eg.
>>
>> kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
>> Topic:redirects PartitionCount:6 ReplicationFactor:2
>> Configs:retention.bytes=10737418240
>> Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2 Isr:
>> 1,2
>> Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0 Isr:
>> 2
>> Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1 Isr:
>> 1
>> Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0 Isr:
>> 1
>> Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1 Isr:
>> 2,1
>> Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2 Isr:
>> 2
>>
>> It run with Zookeeper 3.4.6.
>>
>> As those clusters are in production, I didn't try to migrate more than 1
>> instance after spotting this ISR problem, and then rollback to the original
>> version 0.10.0.1.
>>
>> Any update about this would be greatly receive.
>>
>> --
>> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>>
>> Thomas Kieffer
>>
>> Senior Linux Systems Administrator
>>
>> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263
>> <+352%20691%20444%20263> | www.olamobile.com
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission, dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipient is prohibited. If you received
>> this in error, please contact the sender and delete the material from any
>> computer.
--
<https://fr.linkedin.com/in/thomas-kieffer-28517324>
Thomas Kieffer
Senior Linux Systems Administrator
Skype: thomas.kieffer.corporate | Phone: (+352) 691444263 |
www.olamobile.com <http://www.olamobile.com>
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Re: Lost ISR when upgrading kafka from 0.10.0.1 to any newer version
like 0.10.1.0 or 0.10.2.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Thomas,
Did you follow the instructions:
https://kafka.apache.org/documentation/#upgrade
Ismael
On Mon, Mar 13, 2017 at 9:43 AM, Thomas KIEFFER <
thomas.kieffer@olamobile.com.invalid> wrote:
> I'm trying to perform an upgrade of 2 kafka cluster of 5 instances, When
> I'm doing the switch between 0.10.0.1 and 0.10.1.0 or 0.10.2.0, I saw that
> ISR is lost when I upgrade one instance. I didn't find out yet anything
> relevant about this problem, logs seems just fine.
> eg.
>
> kafka-topics.sh --describe --zookeeper kazoo002.#####.prv --topic redirects
> Topic:redirects PartitionCount:6 ReplicationFactor:2
> Configs:retention.bytes=10737418240
> Topic: redirects Partition: 0 Leader: 1 Replicas: 1,2 Isr:
> 1,2
> Topic: redirects Partition: 1 Leader: 2 Replicas: 2,0 Isr:
> 2
> Topic: redirects Partition: 2 Leader: 1 Replicas: 0,1 Isr:
> 1
> Topic: redirects Partition: 3 Leader: 1 Replicas: 1,0 Isr:
> 1
> Topic: redirects Partition: 4 Leader: 2 Replicas: 2,1 Isr:
> 2,1
> Topic: redirects Partition: 5 Leader: 2 Replicas: 0,2 Isr:
> 2
>
> It run with Zookeeper 3.4.6.
>
> As those clusters are in production, I didn't try to migrate more than 1
> instance after spotting this ISR problem, and then rollback to the original
> version 0.10.0.1.
>
> Any update about this would be greatly receive.
>
> --
> <https://fr.linkedin.com/in/thomas-kieffer-28517324>
>
> Thomas Kieffer
>
> Senior Linux Systems Administrator
>
> Skype: thomas.kieffer.corporate | Phone: (+352) 691444263
> <+352%20691%20444%20263> | www.olamobile.com
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.