Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

[Ismael Juma <is...@juma.me.uk>]

I'm +1 for this change, but we should do it early in the release cycle.
Perhaps 3.6.0 is the right release target. That should buy us enough time
for users to migrate to kraft mode.

Ismael

On Mon, Mar 20, 2023 at 10:12 AM Divij Vaidya <di...@gmail.com>
wrote:

> Hey Colin
>
> Thank you for your feedback. In addition to what Christo mentioned above, I
> have tried to provide answers to your questions below. Also, for some
> context, we have had some conversation about the upgrade in the comments of
> this PR <
> https://github.com/apache/kafka/pull/12620#issuecomment-1409015865>
> .
>
> #1 We shouldn't drop support for rolling upgrades
>
> We are not dropping support for rolling upgrades. Christo's answer above
> hopefully resolves that concern.
>
> #2 Unless there is a security issue, we shouldn't upgrade Zk since Kafka
> 4.0 is going to remove the component
>
> First, if a zero-day exploit/vulnerability is discovered, Zk will not
> backport it to Zookeeper 3.6.4 since it has declared it as end of life. At
> that stage, we will either have to backport the fix to Zk 3.6.4 ourselves
> OR we will have to ask our users to upgrade to Zookeeper 3.8.x at a very
> short notice. Both the options are highly undesirable in my opinion.
>
> Second, even without a vulnerability, many compliance programs red flags
> usage of end of life software. Users of Kafka may be in violation of
> compliance even if they are using the latest version of Kafka (3.5) due to
> the Zookeeper dependency.
>
> Third, the community hasn't decided on a date for 4.0 release. Looking at
> the body of work required to migrate to 4.0, I would say (again, please
> correct me here if you think otherwise) it's at 12 months down the line. I
> think that is a long time to have users of Kafka facing compliance
> violations and at the risk of security exploits.
>
> #3 Major Zk upgrade is risky and may produce bugs
>
> Me and Christo are happy to perform any de-risking activities that you
> would recommend to us, in addition to what we have added in the KIP. I
> think it is worth the investment for the community due to Zookeeper removal
> being far ahead down the line.
>
> --
> Divij Vaidya
>
>
>
> On Wed, Mar 15, 2023 at 12:59 PM Christo Lolov <ch...@gmail.com>
> wrote:
>
> > Hello Colin,
> >
> > Thank you for taking the time to review the proposal!
> >
> > I have attached a compatibility matrix to aid the explanation below - if
> > the mailing system rejects it I will find another way to share it.
> >
> > For the avoidance of doubt, I am not proposing to drop support for
> rolling
> > upgrade from old Kafka versions to new ones. What I am saying is that
> > additional care will need to be taken when upgrading to the latest Kafka
> > versions depending on the version of the accompanying Zookeeper cluster.
> > This additional care means one might have to upgrade to a Kafka version
> > which falls in the intersection of the two sets in the accompanying
> diagram
> > before upgrading the accompanying Zookeeper cluster.
> >
> > As a concrete example let's say you want to upgrade to Kafka 3.5 from
> > Kafka 2.3 and Zookeeper 3.4. You will have to:
> > 1. Carry out a rolling upgrade of your Kafka cluster to a version between
> > 2.4 and 3.4.
> > 2. Carry out a rolling upgrade of your Zookeeper cluster to 3.8.1 (with a
> > possible stop at 3.4.6 due to
> >
> https://zookeeper.apache.org/doc/r3.8.1/zookeeperReconfig.html#ch_reconfig_upgrade
> > ).
> > 3. Carry out a rolling upgrade of your Kafka cluster from 3.4 to 3.5.
> >
> > It is true that Zookeeper is to be deprecated in Kafka 4.0, but as far as
> > I looked there is no concrete release date for that version yet. Until
> this
> > is the case and unless we carry out a Zookeeper version upgrade we leave
> > users to run on an end-of-life version with unpatched CVEs addressed in
> > later versions. Some users have compliance requirements to only run on
> > stable versions of a software and its dependencies and not keeping the
> > dependencies up to date renders them unable to use Kafka.
> >
> > Please, let me know your thoughts on the matter!
> >
> > Best,
> > Christo
> >
> > On Thu, 9 Mar 2023 at 21:56, Colin McCabe <cm...@apache.org> wrote:
> >
> >> Hi,
> >>
> >> I'm struggling a bit with this KIP, because dropping support for rolling
> >> upgrades from old Kafka versions doesn't seem like something we should
> do
> >> in a minor release. But on the other hand, the next Kafka release won't
> >> have ZK at all. Maybe we should punt on this until and unless there is a
> >> security issue that requires some action from us.
> >>
> >> I would also add, that a major ZK version bump is pretty risky. Last
> time
> >> we did this we hit several bugs. I remember we hit one where there was
> an
> >> incompatible change with regard to formatting (sorry, I can't seem to
> find
> >> the JIRA right now).
> >>
> >> Sorry, but for now I have to vote -1 until I can understand this better
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Feb 23, 2023, at 06:48, Divij Vaidya wrote:
> >> > Thanks for the KIP Christo.
> >> >
> >> > Having Zk 3.6 reach EOL in Dec 2022 is a good enough reason to
> upgrade,
> >> > hence I completely agree with the motivation. Your experiments have
> >> > demonstrated that the new version of Zk is stable at scale and the
> >> backward
> >> > compatibility risks are acceptable since Apache Kafka 2.4.x is an EOL
> >> > version.
> >> >
> >> > Vote +1 (non binding)
> >> >
> >> > --
> >> > Divij Vaidya
> >> >
> >> >
> >> >
> >> > On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov <christololov@gmail.com
> >
> >> > wrote:
> >> >
> >> >> Hello!
> >> >>
> >> >> I would like to start the vote for KIP-902, which upgrades Zookeeper
> to
> >> >> version 3.8.1.
> >> >>
> >> >> The KIP can be found at
> >> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
> >> >>
> >> >> The discussion thread is
> >> >> https://lists.apache.org/thread/5jbn2x0rtmqz5scyoygbdbj4vo0mpbw1
> >> >>
> >> >> Thanks
> >> >> Christo
> >>
> >
>