You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Christo Lolov <ch...@gmail.com> on 2023/02/23 14:32:12 UTC

[VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Hello!

I would like to start the vote for KIP-902, which upgrades Zookeeper to version 3.8.1.

The KIP can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784

The discussion thread is https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1

Thanks
Christo

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Ismael Juma <is...@juma.me.uk>.
I'm +1 for this change, but we should do it early in the release cycle.
Perhaps 3.6.0 is the right release target. That should buy us enough time
for users to migrate to kraft mode.

Ismael

On Mon, Mar 20, 2023 at 10:12 AM Divij Vaidya <di...@gmail.com>
wrote:

> Hey Colin
>
> Thank you for your feedback. In addition to what Christo mentioned above, I
> have tried to provide answers to your questions below. Also, for some
> context, we have had some conversation about the upgrade in the comments of
> this PR <
> https://github.com/apache/kafka/pull/12620#issuecomment-1409015865>
> .
>
> #1 We shouldn't drop support for rolling upgrades
>
> We are not dropping support for rolling upgrades. Christo's answer above
> hopefully resolves that concern.
>
> #2 Unless there is a security issue, we shouldn't upgrade Zk since Kafka
> 4.0 is going to remove the component
>
> First, if a zero-day exploit/vulnerability is discovered, Zk will not
> backport it to Zookeeper 3.6.4 since it has declared it as end of life. At
> that stage, we will either have to backport the fix to Zk 3.6.4 ourselves
> OR we will have to ask our users to upgrade to Zookeeper 3.8.x at a very
> short notice. Both the options are highly undesirable in my opinion.
>
> Second, even without a vulnerability, many compliance programs red flags
> usage of end of life software. Users of Kafka may be in violation of
> compliance even if they are using the latest version of Kafka (3.5) due to
> the Zookeeper dependency.
>
> Third, the community hasn't decided on a date for 4.0 release. Looking at
> the body of work required to migrate to 4.0, I would say (again, please
> correct me here if you think otherwise) it's at 12 months down the line. I
> think that is a long time to have users of Kafka facing compliance
> violations and at the risk of security exploits.
>
> #3 Major Zk upgrade is risky and may produce bugs
>
> Me and Christo are happy to perform any de-risking activities that you
> would recommend to us, in addition to what we have added in the KIP. I
> think it is worth the investment for the community due to Zookeeper removal
> being far ahead down the line.
>
> --
> Divij Vaidya
>
>
>
> On Wed, Mar 15, 2023 at 12:59 PM Christo Lolov <ch...@gmail.com>
> wrote:
>
> > Hello Colin,
> >
> > Thank you for taking the time to review the proposal!
> >
> > I have attached a compatibility matrix to aid the explanation below - if
> > the mailing system rejects it I will find another way to share it.
> >
> > For the avoidance of doubt, I am not proposing to drop support for
> rolling
> > upgrade from old Kafka versions to new ones. What I am saying is that
> > additional care will need to be taken when upgrading to the latest Kafka
> > versions depending on the version of the accompanying Zookeeper cluster.
> > This additional care means one might have to upgrade to a Kafka version
> > which falls in the intersection of the two sets in the accompanying
> diagram
> > before upgrading the accompanying Zookeeper cluster.
> >
> > As a concrete example let's say you want to upgrade to Kafka 3.5 from
> > Kafka 2.3 and Zookeeper 3.4. You will have to:
> > 1. Carry out a rolling upgrade of your Kafka cluster to a version between
> > 2.4 and 3.4.
> > 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with a
> > possible stop at 3.4.6 due to
> >
> https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade
> > ).
> > 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.
> >
> > It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far as
> > I looked there is no concrete release date for that version yet. Until
> this
> > is the case and unless we carry out a Zookeeper version upgrade we leave
> > users to run on an end-of-life version with unpatched CVEs addressed in
> > later versions. Some users have compliance requirements to only run on
> > stable versions of a software and its dependencies and not keeping the
> > dependencies up to date renders them unable to use Kafka.
> >
> > Please, let me know your thoughts on the matter!
> >
> > Best,
> > Christo
> >
> > On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:
> >
> >> Hi,
> >>
> >> I'm struggling a bit with this KIP, because dropping support for rolling
> >> upgrades from old Kafka versions doesn't seem like something we should
> do
> >> in a minor release. But on the other hand, the next Kafka release won't
> >> have ZK at all. Maybe we should punt on this until and unless there is a
> >> security issue that requires some action from us.
> >>
> >> I would also add, that a major ZK version bump is pretty risky. Last
> time
> >> we did this we hit several bugs. I remember we hit one where there was
> an
> >> incompatible change with regard to formatting (sorry, I can't seem to
> find
> >> the JIRA right now).
> >>
> >> Sorry, but for now I have to vote -1 until I can understand this better
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
> >> > Thanks for the KIP Christo.
> >> >
> >> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to
> upgrade,
> >> > hence I completely agree with the motivation. Your experiments have
> >> > demonstrated that the new version of Zk is stable at scale and the
> >> backward
> >> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
> >> > version.
> >> >
> >> > Vote +1 (non binding)
> >> >
> >> > --
> >> > Divij Vaidya
> >> >
> >> >
> >> >
> >> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <christololov@gmail.com
> >
> >> > wrote:
> >> >
> >> >> Hello!
> >> >>
> >> >> I would like to start the vote for KIP-902, which upgrades Zookeeper
> to
> >> >> version 3.8.1.
> >> >>
> >> >> The KIP can be found at
> >> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
> >> >>
> >> >> The discussion thread is
> >> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
> >> >>
> >> >> Thanks
> >> >> Christo
> >>
> >
>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Divij Vaidya <di...@gmail.com>.
Hey Colin

Thank you for your feedback. In addition to what Christo mentioned above, I
have tried to provide answers to your questions below. Also, for some
context, we have had some conversation about the upgrade in the comments of
this PR <https://github.com/apache/kafka/pull/12620#issuecomment-1409015865>
.

#1 We shouldn't drop support for rolling upgrades

We are not dropping support for rolling upgrades. Christo's answer above
hopefully resolves that concern.

#2 Unless there is a security issue, we shouldn't upgrade Zk since Kafka
4.0 is going to remove the component

First, if a zero-day exploit/vulnerability is discovered, Zk will not
backport it to Zookeeper 3.6.4 since it has declared it as end of life. At
that stage, we will either have to backport the fix to Zk 3.6.4 ourselves
OR we will have to ask our users to upgrade to Zookeeper 3.8.x at a very
short notice. Both the options are highly undesirable in my opinion.

Second, even without a vulnerability, many compliance programs red flags
usage of end of life software. Users of Kafka may be in violation of
compliance even if they are using the latest version of Kafka (3.5) due to
the Zookeeper dependency.

Third, the community hasn't decided on a date for 4.0 release. Looking at
the body of work required to migrate to 4.0, I would say (again, please
correct me here if you think otherwise) it's at 12 months down the line. I
think that is a long time to have users of Kafka facing compliance
violations and at the risk of security exploits.

#3 Major Zk upgrade is risky and may produce bugs

Me and Christo are happy to perform any de-risking activities that you
would recommend to us, in addition to what we have added in the KIP. I
think it is worth the investment for the community due to Zookeeper removal
being far ahead down the line.

--
Divij Vaidya



On Wed, Mar 15, 2023 at 12:59 PM Christo Lolov <ch...@gmail.com>
wrote:

> Hello Colin,
>
> Thank you for taking the time to review the proposal!
>
> I have attached a compatibility matrix to aid the explanation below - if
> the mailing system rejects it I will find another way to share it.
>
> For the avoidance of doubt, I am not proposing to drop support for rolling
> upgrade from old Kafka versions to new ones. What I am saying is that
> additional care will need to be taken when upgrading to the latest Kafka
> versions depending on the version of the accompanying Zookeeper cluster.
> This additional care means one might have to upgrade to a Kafka version
> which falls in the intersection of the two sets in the accompanying diagram
> before upgrading the accompanying Zookeeper cluster.
>
> As a concrete example let's say you want to upgrade to Kafka 3.5 from
> Kafka 2.3 and Zookeeper 3.4. You will have to:
> 1. Carry out a rolling upgrade of your Kafka cluster to a version between
> 2.4 and 3.4.
> 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with a
> possible stop at 3.4.6 due to
> https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade
> ).
> 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.
>
> It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far as
> I looked there is no concrete release date for that version yet. Until this
> is the case and unless we carry out a Zookeeper version upgrade we leave
> users to run on an end-of-life version with unpatched CVEs addressed in
> later versions. Some users have compliance requirements to only run on
> stable versions of a software and its dependencies and not keeping the
> dependencies up to date renders them unable to use Kafka.
>
> Please, let me know your thoughts on the matter!
>
> Best,
> Christo
>
> On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:
>
>> Hi,
>>
>> I'm struggling a bit with this KIP, because dropping support for rolling
>> upgrades from old Kafka versions doesn't seem like something we should do
>> in a minor release. But on the other hand, the next Kafka release won't
>> have ZK at all. Maybe we should punt on this until and unless there is a
>> security issue that requires some action from us.
>>
>> I would also add, that a major ZK version bump is pretty risky. Last time
>> we did this we hit several bugs. I remember we hit one where there was an
>> incompatible change with regard to formatting (sorry, I can't seem to find
>> the JIRA right now).
>>
>> Sorry, but for now I have to vote -1 until I can understand this better
>>
>> best,
>> Colin
>>
>>
>> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
>> > Thanks for the KIP Christo.
>> >
>> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
>> > hence I completely agree with the motivation. Your experiments have
>> > demonstrated that the new version of Zk is stable at scale and the
>> backward
>> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
>> > version.
>> >
>> > Vote +1 (non binding)
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <ch...@gmail.com>
>> > wrote:
>> >
>> >> Hello!
>> >>
>> >> I would like to start the vote for KIP-902, which upgrades Zookeeper to
>> >> version 3.8.1.
>> >>
>> >> The KIP can be found at
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>> >>
>> >> The discussion thread is
>> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
>> >>
>> >> Thanks
>> >> Christo
>>
>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Colin McCabe <cm...@apache.org>.
On Tue, Apr 18, 2023, at 09:01, Christo Lolov wrote:
> Thank you all for the suggestions for improvements on the KIP and the
> resulting discussions!
>
> Since the voting has been opened for ~2 months and I have received 3
> binding +1 (Colin, Ismael, Mickael) and 1 non-binding +1 (Divij) I will
> move the KIP to accepted and update the associated pull request so that it
> is ready to merge in trunk as soon as 3.5 has been made public.

Hi Christo,

Sounds good. We will also want to update the documentation in that PR.

>
> Re: Colin,
>
> What I meant to demonstrate by putting the Kafka clients is the upgrade
> path for people using the Kafka command line tools and the --zookeeper flag
> which was marked as deprecated in Kafka 2.4. I am happy to remove it if you
> think it is unnecessary or amend it so that it becomes clearer. Please let
> me know what you think!

Yes, please remove it.

thanks,
Colin

>
> Best,
> Christo
>
> On Mon, 17 Apr 2023 at 16:30, Mickael Maison <mi...@gmail.com>
> wrote:
>
>> Hi Christo,
>>
>> +1 (binding)
>>
>> Thanks for the KIP
>>
>> On Fri, Apr 14, 2023 at 7:32 PM Colin McCabe <cm...@apache.org> wrote:
>> >
>> > On Sun, Apr 9, 2023, at 19:17, Ismael Juma wrote:
>> > >
>> > > On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> > >
>> > >> We are going to deprecate ZK mode soon. So if this is indeed a
>> requirement
>> > >> (no deprecated software in prod), perhaps those users will have to
>> move to
>> > >> KRaft mode. (Independently of what we decide here)
>> > >>
>> > >
>> > > Not sure where "no deprecated software in prod" is coming from. The
>> concern
>> > > is regarding end-of-life software - i.e. software that no longer
>> receives
>> > > security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
>> > > position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If
>> it's a
>> > > serious security problem, then it's likely that an additional release
>> of
>> > > ZooKeeper 3.6.x might be released. But the more likely case is that a
>> > > library dependency will have a CVE that will trigger the compliance
>> checks
>> > > from enterprise users, but not warrant another ZooKeeper 3.6.x release.
>> >
>> > Hi Ismael,
>> >
>> > Fair enough. There is a difference between deprecated and unsupported.
>> ZK 3.6.x is unsupported which is worse than deprecated, since it means it
>> will not be updated.
>> >
>> > Overall, I agree with you that we're going to have to move to the new
>> version of ZK. This fits in with the overall timeline of one more year of
>> Kafka releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need
>> to be getting security updates for ZK during this time.
>> >
>> > On Wed, Apr 12, 2023, at 08:45, Christo Lolov wrote:
>> > > Hello Colin,
>> > >
>> > > Thank you for the response!
>> > >
>> > > 1. I have attached the compatibility matrix in the KIP under the
>> section
>> > > Compatibility, Deprecation, and Migration Plan.
>> >
>> > Hi Christo,
>> >
>> > Thanks for attaching the matrix to the KIP.
>> >
>> > I don't understand why Kafka clients are part of this matrix. The Kafka
>> client doesn't use ZK directly. (Well, certain very ancient pre-1.0 Kafka
>> clients did, but that was a long time ago). So let's remove this.
>> >
>> > If I understand this correctly, the main documentation that will be
>> needed is for pre-2.4 Kafka releases. Assuming they keep everything "stock"
>> (which in my experience most users do), the net-net is that pre-2.4
>> releases need to make an extra hop through a post-2.4, pre-3.6 release. We
>> will have to document that as prominently as we can.
>> >
>> > I am +1 for this with the proviso that we do it in 3.6. We should update
>> the version as soon as we can post-3.5 so that any bugs shake out as soon
>> as possible.
>> >
>> > best,
>> > Colin
>>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Christo Lolov <ch...@gmail.com>.
Thank you all for the suggestions for improvements on the KIP and the
resulting discussions!

Since the voting has been opened for ~2 months and I have received 3
binding +1 (Colin, Ismael, Mickael) and 1 non-binding +1 (Divij) I will
move the KIP to accepted and update the associated pull request so that it
is ready to merge in trunk as soon as 3.5 has been made public.

Re: Colin,

What I meant to demonstrate by putting the Kafka clients is the upgrade
path for people using the Kafka command line tools and the --zookeeper flag
which was marked as deprecated in Kafka 2.4. I am happy to remove it if you
think it is unnecessary or amend it so that it becomes clearer. Please let
me know what you think!

Best,
Christo

On Mon, 17 Apr 2023 at 16:30, Mickael Maison <mi...@gmail.com>
wrote:

> Hi Christo,
>
> +1 (binding)
>
> Thanks for the KIP
>
> On Fri, Apr 14, 2023 at 7:32 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > On Sun, Apr 9, 2023, at 19:17, Ismael Juma wrote:
> > >
> > > On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > >> We are going to deprecate ZK mode soon. So if this is indeed a
> requirement
> > >> (no deprecated software in prod), perhaps those users will have to
> move to
> > >> KRaft mode. (Independently of what we decide here)
> > >>
> > >
> > > Not sure where "no deprecated software in prod" is coming from. The
> concern
> > > is regarding end-of-life software - i.e. software that no longer
> receives
> > > security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
> > > position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If
> it's a
> > > serious security problem, then it's likely that an additional release
> of
> > > ZooKeeper 3.6.x might be released. But the more likely case is that a
> > > library dependency will have a CVE that will trigger the compliance
> checks
> > > from enterprise users, but not warrant another ZooKeeper 3.6.x release.
> >
> > Hi Ismael,
> >
> > Fair enough. There is a difference between deprecated and unsupported.
> ZK 3.6.x is unsupported which is worse than deprecated, since it means it
> will not be updated.
> >
> > Overall, I agree with you that we're going to have to move to the new
> version of ZK. This fits in with the overall timeline of one more year of
> Kafka releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need
> to be getting security updates for ZK during this time.
> >
> > On Wed, Apr 12, 2023, at 08:45, Christo Lolov wrote:
> > > Hello Colin,
> > >
> > > Thank you for the response!
> > >
> > > 1. I have attached the compatibility matrix in the KIP under the
> section
> > > Compatibility, Deprecation, and Migration Plan.
> >
> > Hi Christo,
> >
> > Thanks for attaching the matrix to the KIP.
> >
> > I don't understand why Kafka clients are part of this matrix. The Kafka
> client doesn't use ZK directly. (Well, certain very ancient pre-1.0 Kafka
> clients did, but that was a long time ago). So let's remove this.
> >
> > If I understand this correctly, the main documentation that will be
> needed is for pre-2.4 Kafka releases. Assuming they keep everything "stock"
> (which in my experience most users do), the net-net is that pre-2.4
> releases need to make an extra hop through a post-2.4, pre-3.6 release. We
> will have to document that as prominently as we can.
> >
> > I am +1 for this with the proviso that we do it in 3.6. We should update
> the version as soon as we can post-3.5 so that any bugs shake out as soon
> as possible.
> >
> > best,
> > Colin
>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Mickael Maison <mi...@gmail.com>.
Hi Christo,

+1 (binding)

Thanks for the KIP

On Fri, Apr 14, 2023 at 7:32 PM Colin McCabe <cm...@apache.org> wrote:
>
> On Sun, Apr 9, 2023, at 19:17, Ismael Juma wrote:
> >
> > On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe <cm...@apache.org> wrote:
> >
> >> We are going to deprecate ZK mode soon. So if this is indeed a requirement
> >> (no deprecated software in prod), perhaps those users will have to move to
> >> KRaft mode. (Independently of what we decide here)
> >>
> >
> > Not sure where "no deprecated software in prod" is coming from. The concern
> > is regarding end-of-life software - i.e. software that no longer receives
> > security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
> > position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If it's a
> > serious security problem, then it's likely that an additional release of
> > ZooKeeper 3.6.x might be released. But the more likely case is that a
> > library dependency will have a CVE that will trigger the compliance checks
> > from enterprise users, but not warrant another ZooKeeper 3.6.x release.
>
> Hi Ismael,
>
> Fair enough. There is a difference between deprecated and unsupported. ZK 3.6.x is unsupported which is worse than deprecated, since it means it will not be updated.
>
> Overall, I agree with you that we're going to have to move to the new version of ZK. This fits in with the overall timeline of one more year of Kafka releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need to be getting security updates for ZK during this time.
>
> On Wed, Apr 12, 2023, at 08:45, Christo Lolov wrote:
> > Hello Colin,
> >
> > Thank you for the response!
> >
> > 1. I have attached the compatibility matrix in the KIP under the section
> > Compatibility, Deprecation, and Migration Plan.
>
> Hi Christo,
>
> Thanks for attaching the matrix to the KIP.
>
> I don't understand why Kafka clients are part of this matrix. The Kafka client doesn't use ZK directly. (Well, certain very ancient pre-1.0 Kafka clients did, but that was a long time ago). So let's remove this.
>
> If I understand this correctly, the main documentation that will be needed is for pre-2.4 Kafka releases. Assuming they keep everything "stock" (which in my experience most users do), the net-net is that pre-2.4 releases need to make an extra hop through a post-2.4, pre-3.6 release. We will have to document that as prominently as we can.
>
> I am +1 for this with the proviso that we do it in 3.6. We should update the version as soon as we can post-3.5 so that any bugs shake out as soon as possible.
>
> best,
> Colin

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Colin McCabe <cm...@apache.org>.
On Sun, Apr 9, 2023, at 19:17, Ismael Juma wrote:
>
> On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe <cm...@apache.org> wrote:
>
>> We are going to deprecate ZK mode soon. So if this is indeed a requirement
>> (no deprecated software in prod), perhaps those users will have to move to
>> KRaft mode. (Independently of what we decide here)
>>
>
> Not sure where "no deprecated software in prod" is coming from. The concern
> is regarding end-of-life software - i.e. software that no longer receives
> security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
> position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If it's a
> serious security problem, then it's likely that an additional release of
> ZooKeeper 3.6.x might be released. But the more likely case is that a
> library dependency will have a CVE that will trigger the compliance checks
> from enterprise users, but not warrant another ZooKeeper 3.6.x release.

Hi Ismael,

Fair enough. There is a difference between deprecated and unsupported. ZK 3.6.x is unsupported which is worse than deprecated, since it means it will not be updated.

Overall, I agree with you that we're going to have to move to the new version of ZK. This fits in with the overall timeline of one more year of Kafka releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need to be getting security updates for ZK during this time.

On Wed, Apr 12, 2023, at 08:45, Christo Lolov wrote:
> Hello Colin,
>
> Thank you for the response!
>
> 1. I have attached the compatibility matrix in the KIP under the section
> Compatibility, Deprecation, and Migration Plan.

Hi Christo,

Thanks for attaching the matrix to the KIP.

I don't understand why Kafka clients are part of this matrix. The Kafka client doesn't use ZK directly. (Well, certain very ancient pre-1.0 Kafka clients did, but that was a long time ago). So let's remove this.

If I understand this correctly, the main documentation that will be needed is for pre-2.4 Kafka releases. Assuming they keep everything "stock" (which in my experience most users do), the net-net is that pre-2.4 releases need to make an extra hop through a post-2.4, pre-3.6 release. We will have to document that as prominently as we can.

I am +1 for this with the proviso that we do it in 3.6. We should update the version as soon as we can post-3.5 so that any bugs shake out as soon as possible.

best,
Colin

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Ismael Juma <is...@juma.me.uk>.
Colin,

See comment below.

On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe <cm...@apache.org> wrote:

> >
> > Until this is the case and unless we carry out a Zookeeper version
> upgrade we leave users to run on an end-of-life version with unpatched CVEs
> addressed in later versions.
> >
> > Some users have compliance requirements to only run on stable versions
> of a software and its dependencies and not keeping the dependencies up to
> date renders them unable to use Kafka.
>
> We are going to deprecate ZK mode soon. So if this is indeed a requirement
> (no deprecated software in prod), perhaps those users will have to move to
> KRaft mode. (Independently of what we decide here)
>

Not sure where "no deprecated software in prod" is coming from. The concern
is regarding end-of-life software - i.e. software that no longer receives
security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If it's a
serious security problem, then it's likely that an additional release of
ZooKeeper 3.6.x might be released. But the more likely case is that a
library dependency will have a CVE that will trigger the compliance checks
from enterprise users, but not warrant another ZooKeeper 3.6.x release.

Ismael

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Christo Lolov <ch...@gmail.com>.
Hello Colin,

Thank you for the response!

1. I have attached the compatibility matrix in the KIP under the section
Compatibility, Deprecation, and Migration Plan.
2. I believe the answer to how many bridge releases (for Kafka) will be
needed to upgrade from 2.0 to 4.0 based on the compatibility matrix is 2 -
one from 2.0 to any of 2.4.x to 3.5.x (now that we are no longer
considering this KIP for 3.5.x) and one from that version to the bridge
release mentioned in KIP-500 and KIP-866 (assuming that bridge release has
a dependency on Zookeeper 3.8.1).
3. What determines whether you need your Zookeeper cluster to first be
upgraded to 3.4.6 is "Upgrading a running ZooKeeper ensemble to 3.5.0
should be done only after upgrading your ensemble to the 3.4.6 release."
(source:
https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade).
Continuing from the example in point 2, since Kafka 2.0 had Zookeeper
3.4.13 no second bridge upgrade for Zookeeper is needed. To clarify, you
would go from Zookeeper 3.4.13 to any between 3.5.x and 3.6.x and then to
3.8.1.
4. Ideally, if users make an error since they will be carrying out a
rolling restart of their Zookeeper cluster the errors should start
appearing with the first Zookeeper instance which is rebooted, so if they
have sufficient monitoring in place they should be able to catch it before
it takes down their whole Kafka cluster. To be honest, I have never had to
downgrade a Zookeeper cluster, but I suspect the procedure is the same as
upgrading but in reverse i.e. stop the new binary, remove the new binary,
put the old binary, start the old binary.
5. Fair point, I meant to say that Zookeeper will no longer be a thing when
Kafka 4.0 arrives.
6. I believe Ismael's response answers your last concern better than I
could.

Best,
Christo

On Mon, 10 Apr 2023 at 00:53, Colin McCabe <cm...@apache.org> wrote:

> On Wed, Mar 15, 2023, at 04:58, Christo Lolov wrote:
> > Hello Colin,
> >
> > Thank you for taking the time to review the proposal!
> >
> > I have attached a compatibility matrix to aid the explanation below - if
> the mailing system rejects it I will find another way to share it.
>
> Hi Christo,
>
> The mailing list doesn't take attachments. So perhaps you could share this
> in textual form?
>
> > For the avoidance of doubt, I am not proposing to drop support for
> rolling upgrade from old Kafka versions to new ones. What I am saying is
> that additional care will need to be taken when upgrading to the latest
> Kafka versions depending on the version of the accompanying Zookeeper
> cluster. This additional care means one might have to upgrade to a Kafka
> version which falls in the intersection of the two sets in the accompanying
> diagram before upgrading the accompanying Zookeeper cluster.
>
> I think we are talking about the same thing, just using different
> terminology. If you have to go through multiple upgrades to get from
> version X to version Y, I would not say there is "support for rolling
> upgrade from X to Y." In particular if you have to go through some other
> version B, I would say that B is the "bridge release."
>
> This is different than having an "upgrade path" -- I think everyone agrees
> that there should be an upgrade path between any two kafka versions (well,
> ones that are 0.8 or newer, at least).
>
> So I'd like to understand what the bridge release would be for this kind
> of change, and how many "hops" would be required to get from, say, 2.0 to
> 4.0. Keeping in mind that 4.0 won't have ZK at all.
>
> > As a concrete example let's say you want to upgrade to Kafka 3.5 from
> Kafka 2.3 and Zookeeper 3.4. You will have to:
> > 1. Carry out a rolling upgrade of your Kafka cluster to a version
> between 2.4 and 3.4.
> > 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with
> a possible stop at 3.4.6 due to
> https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade
> ).
>
> Hmm, what determines whether I have to make the stop or not?
>
> One thing we haven't discussed in this thread is that a lot of users don't
> upgrade ZK when they do a Kafka upgrade. So I'd also like to understand in
> what situations ZK upgrades would be required as part of Kafka upgrades, if
> we bump this version. Also, what will happen if they forget? I assume the
> cluster would be down for a while. Does ZK have a downgrade procedure?
>
> > 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.
> >
> > It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far
> as I looked there is no concrete release date for that version yet.
>
> ZK is not going to be deprecated in AK 4.0, but removed in 4.0.
>
> >
> > Until this is the case and unless we carry out a Zookeeper version
> upgrade we leave users to run on an end-of-life version with unpatched CVEs
> addressed in later versions.
> >
> > Some users have compliance requirements to only run on stable versions
> of a software and its dependencies and not keeping the dependencies up to
> date renders them unable to use Kafka.
>
> We are going to deprecate ZK mode soon. So if this is indeed a requirement
> (no deprecated software in prod), perhaps those users will have to move to
> KRaft mode. (Independently of what we decide here)
>
> best,
> Colin
>
> > Please, let me know your thoughts on the matter!
> >
> > Best,
> > Christo
> >
> > On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:
> >> Hi,
> >>
> >> I'm struggling a bit with this KIP, because dropping support for
> rolling upgrades from old Kafka versions doesn't seem like something we
> should do in a minor release. But on the other hand, the next Kafka release
> won't have ZK at all. Maybe we should punt on this until and unless there
> is a security issue that requires some action from us.
> >>
> >> I would also add, that a major ZK version bump is pretty risky. Last
> time we did this we hit several bugs. I remember we hit one where there was
> an incompatible change with regard to formatting (sorry, I can't seem to
> find the JIRA right now).
> >>
> >> Sorry, but for now I have to vote -1 until I can understand this better
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
> >> > Thanks for the KIP Christo.
> >> >
> >> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to
> upgrade,
> >> > hence I completely agree with the motivation. Your experiments have
> >> > demonstrated that the new version of Zk is stable at scale and the
> backward
> >> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
> >> > version.
> >> >
> >> > Vote +1 (non binding)
> >> >
> >> > --
> >> > Divij Vaidya
> >> >
> >> >
> >> >
> >> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <christololov@gmail.com
> >
> >> > wrote:
> >> >
> >> >> Hello!
> >> >>
> >> >> I would like to start the vote for KIP-902, which upgrades Zookeeper
> to
> >> >> version 3.8.1.
> >> >>
> >> >> The KIP can be found at
> >> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
> >> >>
> >> >> The discussion thread is
> >> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
> >> >>
> >> >> Thanks
> >> >> Christo
>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Colin McCabe <cm...@apache.org>.
On Wed, Mar 15, 2023, at 04:58, Christo Lolov wrote:
> Hello Colin,
> 
> Thank you for taking the time to review the proposal!
> 
> I have attached a compatibility matrix to aid the explanation below - if the mailing system rejects it I will find another way to share it.

Hi Christo,

The mailing list doesn't take attachments. So perhaps you could share this in textual form?

> For the avoidance of doubt, I am not proposing to drop support for rolling upgrade from old Kafka versions to new ones. What I am saying is that additional care will need to be taken when upgrading to the latest Kafka versions depending on the version of the accompanying Zookeeper cluster. This additional care means one might have to upgrade to a Kafka version which falls in the intersection of the two sets in the accompanying diagram before upgrading the accompanying Zookeeper cluster.

I think we are talking about the same thing, just using different terminology. If you have to go through multiple upgrades to get from version X to version Y, I would not say there is "support for rolling upgrade from X to Y." In particular if you have to go through some other version B, I would say that B is the "bridge release."

This is different than having an "upgrade path" -- I think everyone agrees that there should be an upgrade path between any two kafka versions (well, ones that are 0.8 or newer, at least).

So I'd like to understand what the bridge release would be for this kind of change, and how many "hops" would be required to get from, say, 2.0 to 4.0. Keeping in mind that 4.0 won't have ZK at all.

> As a concrete example let's say you want to upgrade to Kafka 3.5 from Kafka 2.3 and Zookeeper 3.4. You will have to:
> 1. Carry out a rolling upgrade of your Kafka cluster to a version between 2.4 and 3.4.
> 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with a possible stop at 3.4.6 due to https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade).

Hmm, what determines whether I have to make the stop or not?

One thing we haven't discussed in this thread is that a lot of users don't upgrade ZK when they do a Kafka upgrade. So I'd also like to understand in what situations ZK upgrades would be required as part of Kafka upgrades, if we bump this version. Also, what will happen if they forget? I assume the cluster would be down for a while. Does ZK have a downgrade procedure?

> 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.
> 
> It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far as I looked there is no concrete release date for that version yet.

ZK is not going to be deprecated in AK 4.0, but removed in 4.0.

> 
> Until this is the case and unless we carry out a Zookeeper version upgrade we leave users to run on an end-of-life version with unpatched CVEs addressed in later versions.
> 
> Some users have compliance requirements to only run on stable versions of a software and its dependencies and not keeping the dependencies up to date renders them unable to use Kafka.

We are going to deprecate ZK mode soon. So if this is indeed a requirement (no deprecated software in prod), perhaps those users will have to move to KRaft mode. (Independently of what we decide here)

best,
Colin

> Please, let me know your thoughts on the matter!
> 
> Best,
> Christo
> 
> On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:
>> Hi,
>> 
>> I'm struggling a bit with this KIP, because dropping support for rolling upgrades from old Kafka versions doesn't seem like something we should do in a minor release. But on the other hand, the next Kafka release won't have ZK at all. Maybe we should punt on this until and unless there is a security issue that requires some action from us.
>> 
>> I would also add, that a major ZK version bump is pretty risky. Last time we did this we hit several bugs. I remember we hit one where there was an incompatible change with regard to formatting (sorry, I can't seem to find the JIRA right now).
>> 
>> Sorry, but for now I have to vote -1 until I can understand this better
>> 
>> best,
>> Colin
>> 
>> 
>> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
>> > Thanks for the KIP Christo.
>> >
>> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
>> > hence I completely agree with the motivation. Your experiments have
>> > demonstrated that the new version of Zk is stable at scale and the backward
>> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
>> > version.
>> >
>> > Vote +1 (non binding)
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <ch...@gmail.com>
>> > wrote:
>> >
>> >> Hello!
>> >>
>> >> I would like to start the vote for KIP-902, which upgrades Zookeeper to
>> >> version 3.8.1.
>> >>
>> >> The KIP can be found at
>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>> >>
>> >> The discussion thread is
>> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
>> >>
>> >> Thanks
>> >> Christo

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Christo Lolov <ch...@gmail.com>.
Hello Colin,

Thank you for taking the time to review the proposal!

I have attached a compatibility matrix to aid the explanation below - if
the mailing system rejects it I will find another way to share it.

For the avoidance of doubt, I am not proposing to drop support for rolling
upgrade from old Kafka versions to new ones. What I am saying is that
additional care will need to be taken when upgrading to the latest Kafka
versions depending on the version of the accompanying Zookeeper cluster.
This additional care means one might have to upgrade to a Kafka version
which falls in the intersection of the two sets in the accompanying diagram
before upgrading the accompanying Zookeeper cluster.

As a concrete example let's say you want to upgrade to Kafka 3.5 from Kafka
2.3 and Zookeeper 3.4. You will have to:
1. Carry out a rolling upgrade of your Kafka cluster to a version between
2.4 and 3.4.
2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with a
possible stop at 3.4.6 due to
https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade
).
3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.

It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far as I
looked there is no concrete release date for that version yet. Until this
is the case and unless we carry out a Zookeeper version upgrade we leave
users to run on an end-of-life version with unpatched CVEs addressed in
later versions. Some users have compliance requirements to only run on
stable versions of a software and its dependencies and not keeping the
dependencies up to date renders them unable to use Kafka.

Please, let me know your thoughts on the matter!

Best,
Christo

On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:

> Hi,
>
> I'm struggling a bit with this KIP, because dropping support for rolling
> upgrades from old Kafka versions doesn't seem like something we should do
> in a minor release. But on the other hand, the next Kafka release won't
> have ZK at all. Maybe we should punt on this until and unless there is a
> security issue that requires some action from us.
>
> I would also add, that a major ZK version bump is pretty risky. Last time
> we did this we hit several bugs. I remember we hit one where there was an
> incompatible change with regard to formatting (sorry, I can't seem to find
> the JIRA right now).
>
> Sorry, but for now I have to vote -1 until I can understand this better
>
> best,
> Colin
>
>
> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
> > Thanks for the KIP Christo.
> >
> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
> > hence I completely agree with the motivation. Your experiments have
> > demonstrated that the new version of Zk is stable at scale and the
> backward
> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
> > version.
> >
> > Vote +1 (non binding)
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <ch...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> I would like to start the vote for KIP-902, which upgrades Zookeeper to
> >> version 3.8.1.
> >>
> >> The KIP can be found at
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
> >>
> >> The discussion thread is
> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
> >>
> >> Thanks
> >> Christo
>

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Colin McCabe <cm...@apache.org>.
Hi,

I'm struggling a bit with this KIP, because dropping support for rolling upgrades from old Kafka versions doesn't seem like something we should do in a minor release. But on the other hand, the next Kafka release won't have ZK at all. Maybe we should punt on this until and unless there is a security issue that requires some action from us.

I would also add, that a major ZK version bump is pretty risky. Last time we did this we hit several bugs. I remember we hit one where there was an incompatible change with regard to formatting (sorry, I can't seem to find the JIRA right now).

Sorry, but for now I have to vote -1 until I can understand this better

best,
Colin


On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
> Thanks for the KIP Christo.
>
> Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
> hence I completely agree with the motivation. Your experiments have
> demonstrated that the new version of Zk is stable at scale and the backward
> compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
> version.
>
> Vote +1 (non binding)
>
> --
> Divij Vaidya
>
>
>
> On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <ch...@gmail.com>
> wrote:
>
>> Hello!
>>
>> I would like to start the vote for KIP-902, which upgrades Zookeeper to
>> version 3.8.1.
>>
>> The KIP can be found at
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>>
>> The discussion thread is
>> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
>>
>> Thanks
>> Christo

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Posted by Divij Vaidya <di...@gmail.com>.
Thanks for the KIP Christo.

Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to upgrade,
hence I completely agree with the motivation. Your experiments have
demonstrated that the new version of Zk is stable at scale and the backward
compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
version.

Vote +1 (non binding)

--
Divij Vaidya



On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <ch...@gmail.com>
wrote:

> Hello!
>
> I would like to start the vote for KIP-902, which upgrades Zookeeper to
> version 3.8.1.
>
> The KIP can be found at
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>
> The discussion thread is
> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
>
> Thanks
> Christo