You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andrey Klochkov <ak...@liftoff.io> on 2020/07/23 20:01:46 UTC
Kafka compatibility with ZK
Hello,
We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
"kafka has switched to the XXX version of ZK" but never says if switching
to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
and ZK compatible?
--
Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Peter Bukowinski <pm...@gmail.com>.
That procedure looks safe and sane to me, Marina.
> On Aug 2, 2020, at 10:04 AM, Marina Popova <pp...@protonmail.com.invalid> wrote:
>
>
> Actually, I'm very interested in your experience as well.... I'm about to start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK 3.5.6
>
> I have Kafka and ZK as separate clusters.
>
> My plan is :
> 1. rolling upgrade the Kafka cluster to 2.4 - using the inter.broker.protocol.version set to 0.11 at first
> 2. rolling upgrade ZK cluster to 3.5.6
> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka cluster again
>
> Anybody sees a problem with this approach?
>
>
> thanks,
> Marina
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io> wrote:
>>
>> Hello,
>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
>> "kafka has switched to the XXX version of ZK" but never says if switching
>> to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
>> and ZK compatible?
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> Andrey Klochkov
>
>
Re: Kafka compatibility with ZK
Posted by Ismael Juma <is...@juma.me.uk>.
Answers inline.
On Fri, Aug 14, 2020 at 6:46 AM Marina Popova
<pp...@protonmail.com.invalid> wrote:
> thank you, Ismael!
> a follow up question (or two :) ):
> -- our clients (consumers and producers) are all upgraded to Kafka 2.4.0 /
> ZK 3.5.6 version at this time, and since it is a very involved process to
> upgrade them, I'd want to leave them at that for now. Any issues you can
> think of running clients at these versions and brokers at version 2.5.1/ZK
> 3.5.8 ?
>
Kafka clients don't use ZK, so we don't have to worry about that aspect.
There are no issues running 2.4.0 clients with 2.5.1 brokers. However,
there have been some important client fixes since 2.4.0 was released so
it's a good idea to upgrade when you have a chance. It can be done
independently of the brokers upgrade,
-- the brokers upgrade from 0.11 directly to 2.5.1: any words of wisdom on
> what particular properties I should pay attention to, that might have
> changed default values or semantics in between? I have compared server,
> consumer, producer and zookeeper properties files between those
> distributions and did not se anything critical, but just wanted to make
> sure....
>
I highly recommend reviewing the "notable changes" section for each release
in the following doc:
https://kafka.apache.org/documentation/#upgrade
A couple of things I would mention:
1. Run the cluster with the existing inter-broker protocol version for a
few days and make sure that all is good with the applications before
bumping it to the current version. You will not be able to downgrade the
inter-broker protocol version after you bump it (the details are mentioned
in the documentation above).
2. The request metrics include a version tag and you may have to adjust
your monitoring tools.
Hope it helps.
Ismael
>
> Thank you!
> Marina
>
> Sent with [ProtonMail](https://protonmail.com) Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, August 11, 2020 6:02 PM, Ismael Juma <is...@juma.me.uk>
> wrote:
>
> > Note that 2.5.1 has now been released.
> >
> > Ismael
> >
> > On Sun, Aug 2, 2020 at 11:13 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> >> Hi Marina,
> >>
> >> If you upgrade to ZK 3.5.x, I highly recommend going with 3.5.8. It has
> a few critical fixes when compared to 3.5.6. Also, Kafka 2.5.1 should be
> out in a few days and I'd recommend that over 2.4.1.
> >>
> >> Ismael
> >>
> >> On Sun, Aug 2, 2020 at 10:04 AM Marina Popova <pp...@protonmail.com.invalid>
> wrote:
> >>
> >>> Actually, I'm very interested in your experience as well.... I'm about
> to start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka
> 2.4/ZK 3.5.6
> >>>
> >>> I have Kafka and ZK as separate clusters.
> >>>
> >>> My plan is :
> >>> 1. rolling upgrade the Kafka cluster to 2.4 - using the
> inter.broker.protocol.version set to 0.11 at first
> >>> 2. rolling upgrade ZK cluster to 3.5.6
> >>> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the
> Kafka cluster again
> >>>
> >>> Anybody sees a problem with this approach?
> >>>
> >>> thanks,
> >>> Marina
> >>>
> >>> Sent with ProtonMail Secure Email.
> >>>
> >>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >>> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <
> aklochkov@liftoff.io> wrote:
> >>>
> >>>> Hello,
> >>>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> needs
> >>>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> that
> >>>> "kafka has switched to the XXX version of ZK" but never says if
> switching
> >>>> to a newer ZK is mandatory or not. What are the guidelines on keeping
> Kafka
> >>>> and ZK compatible?
> >>>>
> >>>>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>>>
> >>>> Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Marina Popova <pp...@protonmail.com.INVALID>.
thank you, Ismael!
a follow up question (or two :) ):
-- our clients (consumers and producers) are all upgraded to Kafka 2.4.0 / ZK 3.5.6 version at this time, and since it is a very involved process to upgrade them, I'd want to leave them at that for now. Any issues you can think of running clients at these versions and brokers at version 2.5.1/ZK 3.5.8 ?
-- the brokers upgrade from 0.11 directly to 2.5.1: any words of wisdom on what particular properties I should pay attention to, that might have changed default values or semantics in between? I have compared server, consumer, producer and zookeeper properties files between those distributions and did not se anything critical, but just wanted to make sure....
Thank you!
Marina
Sent with [ProtonMail](https://protonmail.com) Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, August 11, 2020 6:02 PM, Ismael Juma <is...@juma.me.uk> wrote:
> Note that 2.5.1 has now been released.
>
> Ismael
>
> On Sun, Aug 2, 2020 at 11:13 AM Ismael Juma <is...@juma.me.uk> wrote:
>
>> Hi Marina,
>>
>> If you upgrade to ZK 3.5.x, I highly recommend going with 3.5.8. It has a few critical fixes when compared to 3.5.6. Also, Kafka 2.5.1 should be out in a few days and I'd recommend that over 2.4.1.
>>
>> Ismael
>>
>> On Sun, Aug 2, 2020 at 10:04 AM Marina Popova <pp...@protonmail.com.invalid> wrote:
>>
>>> Actually, I'm very interested in your experience as well.... I'm about to start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK 3.5.6
>>>
>>> I have Kafka and ZK as separate clusters.
>>>
>>> My plan is :
>>> 1. rolling upgrade the Kafka cluster to 2.4 - using the inter.broker.protocol.version set to 0.11 at first
>>> 2. rolling upgrade ZK cluster to 3.5.6
>>> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka cluster again
>>>
>>> Anybody sees a problem with this approach?
>>>
>>> thanks,
>>> Marina
>>>
>>> Sent with ProtonMail Secure Email.
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io> wrote:
>>>
>>>> Hello,
>>>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
>>>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
>>>> "kafka has switched to the XXX version of ZK" but never says if switching
>>>> to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
>>>> and ZK compatible?
>>>>
>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Ismael Juma <is...@juma.me.uk>.
Note that 2.5.1 has now been released.
Ismael
On Sun, Aug 2, 2020 at 11:13 AM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Marina,
>
> If you upgrade to ZK 3.5.x, I highly recommend going with 3.5.8. It has a
> few critical fixes when compared to 3.5.6. Also, Kafka 2.5.1 should be out
> in a few days and I'd recommend that over 2.4.1.
>
> Ismael
>
> On Sun, Aug 2, 2020 at 10:04 AM Marina Popova
> <pp...@protonmail.com.invalid> wrote:
>
>>
>> Actually, I'm very interested in your experience as well.... I'm about to
>> start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK
>> 3.5.6
>>
>> I have Kafka and ZK as separate clusters.
>>
>> My plan is :
>> 1. rolling upgrade the Kafka cluster to 2.4 - using the
>> inter.broker.protocol.version set to 0.11 at first
>> 2. rolling upgrade ZK cluster to 3.5.6
>> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka
>> cluster again
>>
>> Anybody sees a problem with this approach?
>>
>>
>> thanks,
>> Marina
>>
>>
>> Sent with ProtonMail Secure Email.
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io>
>> wrote:
>>
>> > Hello,
>> > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
>> needs
>> > to be upgraded too (we're currently on 3.4.6). The upgrade guide says
>> that
>> > "kafka has switched to the XXX version of ZK" but never says if
>> switching
>> > to a newer ZK is mandatory or not. What are the guidelines on keeping
>> Kafka
>> > and ZK compatible?
>> >
>> >
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> >
>> > Andrey Klochkov
>>
>>
>>
Re: Kafka compatibility with ZK
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Marina,
If you upgrade to ZK 3.5.x, I highly recommend going with 3.5.8. It has a
few critical fixes when compared to 3.5.6. Also, Kafka 2.5.1 should be out
in a few days and I'd recommend that over 2.4.1.
Ismael
On Sun, Aug 2, 2020 at 10:04 AM Marina Popova
<pp...@protonmail.com.invalid> wrote:
>
> Actually, I'm very interested in your experience as well.... I'm about to
> start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK
> 3.5.6
>
> I have Kafka and ZK as separate clusters.
>
> My plan is :
> 1. rolling upgrade the Kafka cluster to 2.4 - using the
> inter.broker.protocol.version set to 0.11 at first
> 2. rolling upgrade ZK cluster to 3.5.6
> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka
> cluster again
>
> Anybody sees a problem with this approach?
>
>
> thanks,
> Marina
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io>
> wrote:
>
> > Hello,
> > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> needs
> > to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> that
> > "kafka has switched to the XXX version of ZK" but never says if switching
> > to a newer ZK is mandatory or not. What are the guidelines on keeping
> Kafka
> > and ZK compatible?
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Andrey Klochkov
>
>
>
Re: Kafka compatibility with ZK
Posted by Ismael Juma <is...@juma.me.uk>.
Great, thanks for the update Manoj and and Andrey! I recommend the simple
upgrade from 2.5.0 to 2.5.1 to get the fix for
https://issues.apache.org/jira/browse/KAFKA-9835.
Ismael
On Thu, Sep 3, 2020 at 9:01 PM <Ma...@cognizant.com> wrote:
> We also upgraded kafka 2.2.1 to kafka 2.5.0 and kept same zookeeper . no
> issued reported .
> Later we also upgraded zookeeper to 3.5.8 . all good .
>
> On 9/3/20, 8:42 PM, "Andrey Klochkov" <ak...@liftoff.io> wrote:
>
> [External]
>
>
> Hello all,
> FWIW we upgraded to Kafka 2.4.1 and kept ZK at 3.4.6, no issues
> noticed.
>
> On Sun, Aug 2, 2020 at 10:04 AM Marina Popova
> <pp...@protonmail.com.invalid> wrote:
>
> >
> > Actually, I'm very interested in your experience as well.... I'm
> about to
> > start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka
> 2.4/ZK
> > 3.5.6
> >
> > I have Kafka and ZK as separate clusters.
> >
> > My plan is :
> > 1. rolling upgrade the Kafka cluster to 2.4 - using the
> > inter.broker.protocol.version set to 0.11 at first
> > 2. rolling upgrade ZK cluster to 3.5.6
> > 3. set inter.broker.protocol.version=2.4.0 and rolling restart the
> Kafka
> > cluster again
> >
> > Anybody sees a problem with this approach?
> >
> >
> > thanks,
> > Marina
> >
> >
> > Sent with ProtonMail Secure Email.
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <
> aklochkov@liftoff.io>
> > wrote:
> >
> > > Hello,
> > > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if
> ZK
> > needs
> > > to be upgraded too (we're currently on 3.4.6). The upgrade guide
> says
> > that
> > > "kafka has switched to the XXX version of ZK" but never says if
> switching
> > > to a newer ZK is mandatory or not. What are the guidelines on
> keeping
> > Kafka
> > > and ZK compatible?
> > >
> > >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >
> > > Andrey Klochkov
> >
> >
> >
>
> --
> Andrey Klochkov
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>
Re: Kafka compatibility with ZK
Posted by Ma...@cognizant.com.
We also upgraded kafka 2.2.1 to kafka 2.5.0 and kept same zookeeper . no issued reported .
Later we also upgraded zookeeper to 3.5.8 . all good .
On 9/3/20, 8:42 PM, "Andrey Klochkov" <ak...@liftoff.io> wrote:
[External]
Hello all,
FWIW we upgraded to Kafka 2.4.1 and kept ZK at 3.4.6, no issues noticed.
On Sun, Aug 2, 2020 at 10:04 AM Marina Popova
<pp...@protonmail.com.invalid> wrote:
>
> Actually, I'm very interested in your experience as well.... I'm about to
> start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK
> 3.5.6
>
> I have Kafka and ZK as separate clusters.
>
> My plan is :
> 1. rolling upgrade the Kafka cluster to 2.4 - using the
> inter.broker.protocol.version set to 0.11 at first
> 2. rolling upgrade ZK cluster to 3.5.6
> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka
> cluster again
>
> Anybody sees a problem with this approach?
>
>
> thanks,
> Marina
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io>
> wrote:
>
> > Hello,
> > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> needs
> > to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> that
> > "kafka has switched to the XXX version of ZK" but never says if switching
> > to a newer ZK is mandatory or not. What are the guidelines on keeping
> Kafka
> > and ZK compatible?
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Andrey Klochkov
>
>
>
--
Andrey Klochkov
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: Kafka compatibility with ZK
Posted by Andrey Klochkov <ak...@liftoff.io>.
Hello all,
FWIW we upgraded to Kafka 2.4.1 and kept ZK at 3.4.6, no issues noticed.
On Sun, Aug 2, 2020 at 10:04 AM Marina Popova
<pp...@protonmail.com.invalid> wrote:
>
> Actually, I'm very interested in your experience as well.... I'm about to
> start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK
> 3.5.6
>
> I have Kafka and ZK as separate clusters.
>
> My plan is :
> 1. rolling upgrade the Kafka cluster to 2.4 - using the
> inter.broker.protocol.version set to 0.11 at first
> 2. rolling upgrade ZK cluster to 3.5.6
> 3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka
> cluster again
>
> Anybody sees a problem with this approach?
>
>
> thanks,
> Marina
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io>
> wrote:
>
> > Hello,
> > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> needs
> > to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> that
> > "kafka has switched to the XXX version of ZK" but never says if switching
> > to a newer ZK is mandatory or not. What are the guidelines on keeping
> Kafka
> > and ZK compatible?
> >
> >
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Andrey Klochkov
>
>
>
--
Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Marina Popova <pp...@protonmail.com.INVALID>.
Actually, I'm very interested in your experience as well.... I'm about to start the same (similar) upgrade - from Kafka 0.11/ZK3.4.13 to Kafka 2.4/ZK 3.5.6
I have Kafka and ZK as separate clusters.
My plan is :
1. rolling upgrade the Kafka cluster to 2.4 - using the inter.broker.protocol.version set to 0.11 at first
2. rolling upgrade ZK cluster to 3.5.6
3. set inter.broker.protocol.version=2.4.0 and rolling restart the Kafka cluster again
Anybody sees a problem with this approach?
thanks,
Marina
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, July 23, 2020 4:01 PM, Andrey Klochkov <ak...@liftoff.io> wrote:
> Hello,
> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
> to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
> "kafka has switched to the XXX version of ZK" but never says if switching
> to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
> and ZK compatible?
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Peter Bukowinski <pm...@gmail.com>.
Zookeeper is not part of the kafka project and must be installed separately. Confluent maintain a version compatibility table you can use as a reference: https://docs.confluent.io/current/installation/versions-interoperability.html#zk <https://docs.confluent.io/current/installation/versions-interoperability.html#zk>
> On Jul 23, 2020, at 1:05 PM, M. Manna <ma...@gmail.com> wrote:
>
> Hi,
>
> AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get what
> is in 2.4.1.
>
> It’s a little different however, if you’re hosting ZK in a different host
> running independently of Kafka.
>
> What’s your situation ?
>
>
>
> On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io> wrote:
>
>> Hello,
>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
>> "kafka has switched to the XXX version of ZK" but never says if switching
>> to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
>> and ZK compatible?
>>
>> --
>> Andrey Klochkov
>>
Re: Kafka compatibility with ZK
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Andrey,
ZK 3.4.6 probably works, but we haven't tested with that version for a
while. Generally, we recommend upgrading to the latest bug fix release of
ZooKeeper (3.4.13, 3.5.8) as they tend to include important fixes. However,
you can do that in your own time, independently of the Kafka upgrade.
Ismael
On Thu, Jul 23, 2020 at 2:21 PM Andrey Klochkov <ak...@liftoff.io>
wrote:
> Hi Peter,
> Thanks, that's very helpful! We'll go on with upgrading to 2.4.1 and not
> touching our ZK 3.4.6 for now. I can report back if that actually causes
> issues just in case anybody is interested.
>
> On Thu, Jul 23, 2020 at 2:00 PM Peter Bukowinski <pm...@gmail.com> wrote:
>
> > Agreed. We use a cloudera distribution of zookeeper, that is versioned at
> > 3.4.5 (plus a bunch of backported patches) with kafka 2.4 and haven’t had
> > any issues.
> >
> >
> > > On Jul 23, 2020, at 1:19 PM, Andrey Klochkov <ak...@liftoff.io>
> > wrote:
> > >
> > > We are running a separate ZK cluster and its version is not really tied
> > to
> > > the version of Kafka we're using.
> > >
> > > I have seen the Confluent compatibility matrix and based on that our
> > > *current* version of Kafka is not compatible with our version of ZK,
> and
> > we
> > > haven't seen any problems with that. My suspicion is that Confluent
> might
> > > be using some of the newer ZK features such as e.g. dynamic
> > > configuration and that's where their requirements come from, but that
> > > doesn't mean Kafka requires the versions of ZK that Confluent lists as
> > > required for the Confluent platform.
> > >
> > > On Thu, Jul 23, 2020 at 1:06 PM M. Manna <ma...@gmail.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get
> > what
> > >> is in 2.4.1.
> > >>
> > >> It’s a little different however, if you’re hosting ZK in a different
> > host
> > >> running independently of Kafka.
> > >>
> > >> What’s your situation ?
> > >>
> > >>
> > >>
> > >> On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io>
> > >> wrote:
> > >>
> > >>> Hello,
> > >>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> > >> needs
> > >>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> > >> that
> > >>> "kafka has switched to the XXX version of ZK" but never says if
> > switching
> > >>> to a newer ZK is mandatory or not. What are the guidelines on keeping
> > >> Kafka
> > >>> and ZK compatible?
> > >>>
> > >>> --
> > >>> Andrey Klochkov
> > >>>
> > >>
> > >
> > >
> > > --
> > > Andrey Klochkov
> >
> >
>
> --
> Andrey Klochkov
>
Re: Kafka compatibility with ZK
Posted by Andrey Klochkov <ak...@liftoff.io>.
Hi Peter,
Thanks, that's very helpful! We'll go on with upgrading to 2.4.1 and not
touching our ZK 3.4.6 for now. I can report back if that actually causes
issues just in case anybody is interested.
On Thu, Jul 23, 2020 at 2:00 PM Peter Bukowinski <pm...@gmail.com> wrote:
> Agreed. We use a cloudera distribution of zookeeper, that is versioned at
> 3.4.5 (plus a bunch of backported patches) with kafka 2.4 and haven’t had
> any issues.
>
>
> > On Jul 23, 2020, at 1:19 PM, Andrey Klochkov <ak...@liftoff.io>
> wrote:
> >
> > We are running a separate ZK cluster and its version is not really tied
> to
> > the version of Kafka we're using.
> >
> > I have seen the Confluent compatibility matrix and based on that our
> > *current* version of Kafka is not compatible with our version of ZK, and
> we
> > haven't seen any problems with that. My suspicion is that Confluent might
> > be using some of the newer ZK features such as e.g. dynamic
> > configuration and that's where their requirements come from, but that
> > doesn't mean Kafka requires the versions of ZK that Confluent lists as
> > required for the Confluent platform.
> >
> > On Thu, Jul 23, 2020 at 1:06 PM M. Manna <ma...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get
> what
> >> is in 2.4.1.
> >>
> >> It’s a little different however, if you’re hosting ZK in a different
> host
> >> running independently of Kafka.
> >>
> >> What’s your situation ?
> >>
> >>
> >>
> >> On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io>
> >> wrote:
> >>
> >>> Hello,
> >>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> >> needs
> >>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> >> that
> >>> "kafka has switched to the XXX version of ZK" but never says if
> switching
> >>> to a newer ZK is mandatory or not. What are the guidelines on keeping
> >> Kafka
> >>> and ZK compatible?
> >>>
> >>> --
> >>> Andrey Klochkov
> >>>
> >>
> >
> >
> > --
> > Andrey Klochkov
>
>
--
Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Peter Bukowinski <pm...@gmail.com>.
Agreed. We use a cloudera distribution of zookeeper, that is versioned at 3.4.5 (plus a bunch of backported patches) with kafka 2.4 and haven’t had any issues.
> On Jul 23, 2020, at 1:19 PM, Andrey Klochkov <ak...@liftoff.io> wrote:
>
> We are running a separate ZK cluster and its version is not really tied to
> the version of Kafka we're using.
>
> I have seen the Confluent compatibility matrix and based on that our
> *current* version of Kafka is not compatible with our version of ZK, and we
> haven't seen any problems with that. My suspicion is that Confluent might
> be using some of the newer ZK features such as e.g. dynamic
> configuration and that's where their requirements come from, but that
> doesn't mean Kafka requires the versions of ZK that Confluent lists as
> required for the Confluent platform.
>
> On Thu, Jul 23, 2020 at 1:06 PM M. Manna <ma...@gmail.com> wrote:
>
>> Hi,
>>
>> AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get what
>> is in 2.4.1.
>>
>> It’s a little different however, if you’re hosting ZK in a different host
>> running independently of Kafka.
>>
>> What’s your situation ?
>>
>>
>>
>> On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io>
>> wrote:
>>
>>> Hello,
>>> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
>> needs
>>> to be upgraded too (we're currently on 3.4.6). The upgrade guide says
>> that
>>> "kafka has switched to the XXX version of ZK" but never says if switching
>>> to a newer ZK is mandatory or not. What are the guidelines on keeping
>> Kafka
>>> and ZK compatible?
>>>
>>> --
>>> Andrey Klochkov
>>>
>>
>
>
> --
> Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by Andrey Klochkov <ak...@liftoff.io>.
We are running a separate ZK cluster and its version is not really tied to
the version of Kafka we're using.
I have seen the Confluent compatibility matrix and based on that our
*current* version of Kafka is not compatible with our version of ZK, and we
haven't seen any problems with that. My suspicion is that Confluent might
be using some of the newer ZK features such as e.g. dynamic
configuration and that's where their requirements come from, but that
doesn't mean Kafka requires the versions of ZK that Confluent lists as
required for the Confluent platform.
On Thu, Jul 23, 2020 at 1:06 PM M. Manna <ma...@gmail.com> wrote:
> Hi,
>
> AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get what
> is in 2.4.1.
>
> It’s a little different however, if you’re hosting ZK in a different host
> running independently of Kafka.
>
> What’s your situation ?
>
>
>
> On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io>
> wrote:
>
> > Hello,
> > We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK
> needs
> > to be upgraded too (we're currently on 3.4.6). The upgrade guide says
> that
> > "kafka has switched to the XXX version of ZK" but never says if switching
> > to a newer ZK is mandatory or not. What are the guidelines on keeping
> Kafka
> > and ZK compatible?
> >
> > --
> > Andrey Klochkov
> >
>
--
Andrey Klochkov
Re: Kafka compatibility with ZK
Posted by "M. Manna" <ma...@gmail.com>.
Hi,
AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get what
is in 2.4.1.
It’s a little different however, if you’re hosting ZK in a different host
running independently of Kafka.
What’s your situation ?
On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov <ak...@liftoff.io> wrote:
> Hello,
> We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs
> to be upgraded too (we're currently on 3.4.6). The upgrade guide says that
> "kafka has switched to the XXX version of ZK" but never says if switching
> to a newer ZK is mandatory or not. What are the guidelines on keeping Kafka
> and ZK compatible?
>
> --
> Andrey Klochkov
>