You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yves Courvoisier <yv...@enata.com> on 2019/12/20 08:59:00 UTC
Re: WELCOME to users@kafka.apache.org
Hi,
Here is a little bit of context: I am part of a development project
where we installed a kafka cluster in our boat. The nature of the
project is hence very different from the initial purpose of kafka,
since we have only one broker per boat that we then replicate on a
master server.
We experienced the following issue trying to upgrade from version 2.3.1
to version 2.4.0:```kafka.common.InconsistentClusterIdException: The
Cluster ID -BMMqSBtSu-WoXyYQcByxw doesn't match stored clusterId
Some(dLIK0LbmTz2wUiTeZ9YWWQ) in meta.properties. The broker is trying
to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44
)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
```
Does this happened to anyone else? (I have found no comments on this)
We saw that an upgrade protocole is detailed on the apache
documentation (https://kafka.apache.org/documentation/#upgrade_2_4_0)
which we applied in a second intent. This did not solve our issue.
Could someone say why such a protocole is necessary?
Where can I find more information on the management of cluster ids and
can we force this id in order to control this does not happen again?
Context:java 13.kafka 2.3.1 -> 2.4running in a docker based on the
image adoptopenjdk:13-jre-hotspot
Thanks for reading.RegardsYves
On Fri, 2019-12-20 at 08:27 +0000, users-help@kafka.apache.org wrote:
> Hi! This is the ezmlm program. I'm managing theusers@kafka.apache.org
> mailing list.
> I'm working for my owner, who can be reachedat
> users-owner@kafka.apache.org.
> Acknowledgment: I have added the address
> yves.courvoisier@enata.com
>
> to the users mailing list.
> Welcome to users@kafka.apache.org!
> Please save this message so that you know the address you
> aresubscribed under, in case you later want to unsubscribe or change
> yoursubscription address.
>
> --- Administrative commands for the users list ---
> I can handle administrative requests automatically. Pleasedo not send
> them to the list address! Instead, sendyour message to the correct
> command address:
> To subscribe to the list, send a message to: <
> users-subscribe@kafka.apache.org>
> To remove your address from the list, send a message to: <
> users-unsubscribe@kafka.apache.org>
> Send mail to the following for info and FAQ for this list: <
> users-info@kafka.apache.org> <us...@kafka.apache.org>
> Similar addresses exist for the digest list: <
> users-digest-subscribe@kafka.apache.org> <
> users-digest-unsubscribe@kafka.apache.org>
> To get messages 123 through 145 (a maximum of 100 per request),
> mail: <us...@kafka.apache.org>
> To get an index with subject and author for messages 123-456 ,
> mail: <us...@kafka.apache.org>
> They are always returned as sets of 100, max 2000 per request,so
> you'll actually get 100-499.
> To receive all messages with the same subject as message 12345,send a
> short message to: <us...@kafka.apache.org>
> The messages should contain one line or word of text to avoid
> beingtreated as sp@m, but I will ignore their content.Only the
> ADDRESS you send to is important.
> You can start a subscription for an alternate address,for example "
> john@host.domain", just add a hyphen and youraddress (with '='
> instead of '@') after the command word:<
> users-subscribe-john=host.domain@kafka.apache.org>
> To stop subscription for this address, mail:<
> users-unsubscribe-john=host.domain@kafka.apache.org>
> In both cases, I'll send a confirmation message to that address.
> Whenyou receive it, simply reply to it to complete your subscription.
> If despite following these instructions, you do not get thedesired
> results, please contact my owner atusers-owner@kafka.apache.org.
> Please be patient, my owner is alot slower than I am ;-)
> --- Enclosed is a copy of the request I received.
> Return-Path: <yv...@enata.com>Received: (qmail 18727
> invoked by uid 99); 20 Dec 2019 08:27:25 -0000Received: from pnap-us-
> west-generic-nat.apache.org (HELO spamd4-us-west.apache.org)
> (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20
> Dec 2019 08:27:25 +0000Received: from localhost (localhost
> [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at
> spamd4-us-west.apache.org) with ESMTP id 963CFC2122 for <
> users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >; Fri, 20 Dec 2019 08:27:24 +0000 (UTC)X-Virus-Scanned: Debian
> amavisd-new at spamd4-us-west.apache.orgX-Spam-Flag: NOX-Spam-Score:
> 1.632X-Spam-Level: *X-Spam-Status: No, score=1.632 tagged_above=-999
> required=6.31 tests=[DKIM_ADSP_ALL=1.1, DKIM_INVALID=0.1,
> DKIM_SIGNED=0.1, HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01,
> SPF_HELO_NONE=0.001, TRACKER_ID=0.1, T_MIME_MALF=0.01,
> T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001]
> autolearn=disabledAuthentication-Results: spamd4-us-west.apache.org
> (amavisd-new); dkim=neutral (2048-bit key) reason="inval
> id (public key: does not support hash algorithm 'sha256')" header.
> d=enata.comReceived: from mx1-he-de.apache.org ([10.40.0.8]) by
> localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port
> 10024) with ESMTP id hGMSySlrQZVu for <
> users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >; Fri, 20 Dec 2019 08:27:22 +0000 (UTC)Received-SPF: Permerror
> (mailfrom) identity=mailfrom; client-ip=99.80.255.160;
> helo=smtp.enata.com; envelope-from=yves.courvoisier@enata.com;
> receiver=<UNKNOWN> Received: from smtp.enata.com (smtp.enata.com
> [99.80.255.160]) by mx1-he-de.apache.org (ASF Mail Server at
> mx1-he-de.apache.org) with ESMTPS id 7A0757DDDD for <
> users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >; Fri, 20 Dec 2019 08:27:21 +0000 (UTC)Received: from
> localhost.localdomain (unknown [192.168.0.1]) (using TLSv1.3 with
> cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate
> requested) by smtp.enata.com (Postfix) with ESMTPSA id C1BC025DAD1
> for <
> users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >; Fri, 20 Dec 2019 08:27:20 +0000 (UTC)DKIM-Signature: v=1; a=rsa-
> sha256; c=simple/simple; d=enata.com; s=enata.com; t=1576830440;
> bh=vCBN8IB6Qt9gG6KH5ZZxAUTjdhREmie4FruUVWzv1N8=; h=Subject:From:
> To:Date:In-Reply-To:References:From; b=AZ4x2zJrw3MQbOKAcudlX0VTZ5lPB
> dB8U8s4Odg95t+8FeobD1yv22kp1e4EToyWw aL1hnC8I9pVDLVvY1rl+IBdMfCeQwM
> pKkwmIig1qaNRLiP1bCdD4T2GF9rI3gwzCZ3 bx68v3mOvFesDho6vJj29OH6KVephp
> a1bLt+MXw2GXnwOmJbRbObxPpH3fRxHLhYEO /z2RzUAzsQg7H6TtGQX/DQILYqU7OF
> mGTAlXKIkD2fJKMVjQwB1B3zcj7JeZ8KKscC sFSy9Wco5YlsSemf4w6ecDPEXKOmD1
> t33nAS5AGsc8sMDt1HYC9Ixib0lHAaREjDgF vY9nIhbeJxG+g==Message-ID: <
> 570200239582fd6fcad0e716a0b2e8b3610244ec.camel@enata.com>Subject: Re:
> confirm subscribe to users@kafka.apache.org
> From: Yves Courvoisier <yv...@enata.com>To:
> users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> Date: Fri, 20 Dec 2019 09:27:17 +0100In-Reply-To: <
> 1576830319.17328.ezmlm@kafka.apache.org>References: <
> 1576830319.17328.ezmlm@kafka.apache.org>Content-Type:
> multipart/alternative; boundary="=-ne3xFoF4/GqgIqU97haH"User-Agent:
> Evolution 3.34.2 (3.34.2-1.fc31) MIME-Version: 1.0
--
Yves Courvoisier
Simulation Software Developer
M: +41 78 898 13 53
T: +41 22 900 00 53
W: www.enata.com
This message, including any attachments may contain confidential and privileged material; it is intended only for the person to whom it is addressed. The contents of this email does not constitute a commitment by Enata Industries FZE except where provided for in a written agreement. Enata Industries FZE assumes no liability or responsibility for the consequences arising out of a delay and/or loss in transit of this message, or for corruption or other error(s) arising in its transmission and for any misuse or fraudulent use which may be made thereof. If you are not the intended recipient, please contact us and abstain from any disclosure, use or dissemination.
Re: kafka upgrade 2.3 -> 2.4 cluster.id issue
Posted by Yves Courvoisier <yv...@enata.com>.
Hi again,
Sorry for the formatting of my mail. I changed to plain text. Below the
same mail but with correct formatting:
Here is a little bit of context: I am part of a development project
where we installed a kafka cluster in our boat. The nature of the
project is hence very different from the initial purpose of kafka,
since we have only one broker per boat that we then replicate on a
master server.
We experienced the following issue trying to upgrade from version 2.3.1
to version 2.4.0:
```
kafka.common.InconsistentClusterIdException: The Cluster ID -BMMqSBtSu-
WoXyYQcByxw doesn't match stored clusterId Some(dLIK0LbmTz2wUiTeZ9YWWQ)
in meta.properties. The broker is trying to join the wrong cluster.
Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44
)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
```
Does this happened to anyone else? (I have found no comments on this)
We saw that an upgrade protocole is detailed on the apache
documentation (https://kafka.apache.org/documentation/#upgrade_2_4_0)
which we applied in a second intent. This did not solve our issue.
Could someone say why such a protocole is necessary?
Where can I find more information on the management of cluster ids and
can we force this id in order to control this does not happen again?
Context:
java 13.
kafka 2.3.1 -> 2.4
running in a docker based on the image adoptopenjdk:13-jre-hotspot
Thanks for reading.
Regards
Yves
On Fri, 2019-12-20 at 09:59 +0100, Yves Courvoisier wrote:
> Hi,
> Here is a little bit of context: I am part of a development project
> where we installed a kafka cluster in our boat. The nature of the
> project is hence very different from the initial purpose of kafka,
> since we have only one broker per boat that we then replicate on a
> master server.
> We experienced the following issue trying to upgrade from version
> 2.3.1
> to version 2.4.0:```kafka.common.InconsistentClusterIdException: The
> Cluster ID -BMMqSBtSu-WoXyYQcByxw doesn't match stored clusterId
> Some(dLIK0LbmTz2wUiTeZ9YWWQ) in meta.properties. The broker is trying
> to join the wrong cluster. Configured zookeeper.connect may be wrong.
> at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
> at
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:
> 44
> )
> at kafka.Kafka$.main(Kafka.scala:84)
> at kafka.Kafka.main(Kafka.scala)
>
> ```
> Does this happened to anyone else? (I have found no comments on this)
> We saw that an upgrade protocole is detailed on the apache
> documentation (
> https://kafka.apache.org/documentation/#upgrade_2_4_0
> )
> which we applied in a second intent. This did not solve our issue.
> Could someone say why such a protocole is necessary?
> Where can I find more information on the management of cluster ids
> and
> can we force this id in order to control this does not happen again?
> Context:java 13.kafka 2.3.1 -> 2.4running in a docker based on the
> image adoptopenjdk:13-jre-hotspot
> Thanks for reading.RegardsYves
>
>
> On Fri, 2019-12-20 at 08:27 +0000,
> users-help@kafka.apache.org
> wrote:
> > Hi! This is the ezmlm program. I'm managing
> > theusers@kafka.apache.org
> >
> > mailing list.
> > I'm working for my owner, who can be reachedat
> > users-owner@kafka.apache.org
> > .
> > Acknowledgment: I have added the address
> >
> > yves.courvoisier@enata.com
> >
> >
> > to the users mailing list.
> > Welcome to
> > users@kafka.apache.org
> > !
> > Please save this message so that you know the address you
> > aresubscribed under, in case you later want to unsubscribe or
> > change
> > yoursubscription address.
> >
> > --- Administrative commands for the users list ---
> > I can handle administrative requests automatically. Pleasedo not
> > send
> > them to the list address! Instead, sendyour message to the correct
> > command address:
> > To subscribe to the list, send a message to: <
> > users-subscribe@kafka.apache.org
> > >
> > To remove your address from the list, send a message to: <
> > users-unsubscribe@kafka.apache.org
> > >
> > Send mail to the following for info and FAQ for this list: <
> > users-info@kafka.apache.org
> > > <
> > users-faq@kafka.apache.org
> > >
> > Similar addresses exist for the digest list: <
> > users-digest-subscribe@kafka.apache.org
> > > <
> > users-digest-unsubscribe@kafka.apache.org
> > >
> > To get messages 123 through 145 (a maximum of 100 per request),
> > mail: <
> > users-get.123_145@kafka.apache.org
> > >
> > To get an index with subject and author for messages 123-456 ,
> > mail: <
> > users-index.123_456@kafka.apache.org
> > >
> > They are always returned as sets of 100, max 2000 per request,so
> > you'll actually get 100-499.
> > To receive all messages with the same subject as message 12345,send
> > a
> > short message to: <
> > users-thread.12345@kafka.apache.org
> > >
> > The messages should contain one line or word of text to avoid
> > beingtreated as sp@m, but I will ignore their content.Only the
> > ADDRESS you send to is important.
> > You can start a subscription for an alternate address,for example "
> > john@host.domain
> > ", just add a hyphen and youraddress (with '='
> > instead of '@') after the command word:<
> > users-subscribe-john=host.domain@kafka.apache.org
> > >
> > To stop subscription for this address, mail:<
> > users-unsubscribe-john=host.domain@kafka.apache.org
> > >
> > In both cases, I'll send a confirmation message to that address.
> > Whenyou receive it, simply reply to it to complete your
> > subscription.
> > If despite following these instructions, you do not get thedesired
> > results, please contact my owner
> > atusers-owner@kafka.apache.org
> > .
> > Please be patient, my owner is alot slower than I am ;-)
> > --- Enclosed is a copy of the request I received.
> > Return-Path: <
> > yves.courvoisier@enata.com
> > >Received: (qmail 18727
> > invoked by uid 99); 20 Dec 2019 08:27:25 -0000Received: from pnap-
> > us-
> > west-generic-nat.apache.org (HELO spamd4-us-west.apache.org)
> > (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri,
> > 20
> > Dec 2019 08:27:25 +0000Received: from localhost (localhost
> > [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server
> > at
> > spamd4-us-west.apache.org) with ESMTP id 963CFC2122 for <
> > users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >
> > > ; Fri, 20 Dec 2019 08:27:24 +0000 (UTC)X-Virus-Scanned: Debian
> >
> > amavisd-new at spamd4-us-west.apache.orgX-Spam-Flag: NOX-Spam-
> > Score:
> > 1.632X-Spam-Level: *X-Spam-Status: No, score=1.632 tagged_above=-
> > 999
> > required=6.31 tests=[DKIM_ADSP_ALL=1.1, DKIM_INVALID=0.1,
> > DKIM_SIGNED=0.1, HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01,
> > SPF_HELO_NONE=0.001, TRACKER_ID=0.1, T_MIME_MALF=0.01,
> > T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001]
> > autolearn=disabledAuthentication-Results: spamd4-us-west.apache.org
> > (amavisd-new); dkim=neutral (2048-bit key) reason="inval
> > id (public key: does not support hash algorithm 'sha256')" header.
> > d=enata.comReceived: from mx1-he-de.apache.org ([10.40.0.8])
> > by
> > localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new,
> > port
> > 10024) with ESMTP id hGMSySlrQZVu for <
> > users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >
> > > ; Fri, 20 Dec 2019 08:27:22 +0000 (UTC)Received-SPF: Permerror
> >
> > (mailfrom) identity=mailfrom; client-ip=99.80.255.160;
> > helo=smtp.enata.com;
> > envelope-from=yves.courvoisier@enata.com
> > ;
> > receiver=<UNKNOWN> Received: from smtp.enata.com (smtp.enata.com
> > [99.80.255.160]) by mx1-he-de.apache.org (ASF Mail Server at
> > mx1-he-de.apache.org) with ESMTPS id 7A0757DDDD for <
> > users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >
> > > ; Fri, 20 Dec 2019 08:27:21 +0000 (UTC)Received: from
> >
> > localhost.localdomain (unknown [192.168.0.1]) (using TLSv1.3
> > with
> > cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client
> > certificate
> > requested) by smtp.enata.com (Postfix) with ESMTPSA id C1BC025DAD1
> > for <
> > users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >
> > > ; Fri, 20 Dec 2019 08:27:20 +0000 (UTC)DKIM-Signature: v=1;
> > > a=rsa-
> >
> > sha256; c=simple/simple; d=enata.com; s=enata.com;
> > t=1576830440;
> > bh=vCBN8IB6Qt9gG6KH5ZZxAUTjdhREmie4FruUVWzv1N8=; h=Subject:From:
> > To:Date:In-Reply-To:References:From; b=AZ4x2zJrw3MQbOKAcudlX
> > 0VTZ5lPB
> > dB8U8s4Odg95t+8FeobD1yv22kp1e4EToyWw aL1hnC8I9pVDLVvY1rl+IB
> > dMfCeQwM
> > pKkwmIig1qaNRLiP1bCdD4T2GF9rI3gwzCZ3 bx68v3mOvFesDho6vJj29O
> > H6KVephp
> > a1bLt+MXw2GXnwOmJbRbObxPpH3fRxHLhYEO /z2RzUAzsQg7H6TtGQX/DQ
> > ILYqU7OF
> > mGTAlXKIkD2fJKMVjQwB1B3zcj7JeZ8KKscC sFSy9Wco5YlsSemf4w6ecD
> > PEXKOmD1
> > t33nAS5AGsc8sMDt1HYC9Ixib0lHAaREjDgF vY9nIhbeJxG+g==Message
> > -ID: <
> > 570200239582fd6fcad0e716a0b2e8b3610244ec.camel@enata.com
> > >Subject: Re:
> > confirm subscribe to
> > users@kafka.apache.org
> >
> > From: Yves Courvoisier <
> > yves.courvoisier@enata.com
> > >To:
> > users-sc.1576830319.pdifjoilmdgohnlibdbm-yves.courvoisier=enata.com@kafka.apache.org
> >
> > Date: Fri, 20 Dec 2019 09:27:17 +0100In-Reply-To: <
> > 1576830319.17328.ezmlm@kafka.apache.org
> > >References: <
> > 1576830319.17328.ezmlm@kafka.apache.org
> > >Content-Type:
> > multipart/alternative; boundary="=-ne3xFoF4/GqgIqU97haH"User-Agent:
> > Evolution 3.34.2 (3.34.2-1.fc31) MIME-Version: 1.0
--
Yves Courvoisier
Simulation Software Developer
M: +41 78 898 13 53
T: +41 22 900 00 53
W: www.enata.com
This message, including any attachments may contain confidential and privileged material; it is intended only for the person to whom it is addressed. The contents of this email does not constitute a commitment by Enata Industries FZE except where provided for in a written agreement. Enata Industries FZE assumes no liability or responsibility for the consequences arising out of a delay and/or loss in transit of this message, or for corruption or other error(s) arising in its transmission and for any misuse or fraudulent use which may be made thereof. If you are not the intended recipient, please contact us and abstain from any disclosure, use or dissemination.