You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Colin McCabe <cm...@apache.org> on 2022/05/04 01:03:20 UTC
[DISCUSS] KIP-833: Mark KRaft as Production Ready
Hi all,
I've written a KIP for marking KRaft as production ready. Please take a look if you have a chance:
https://cwiki.apache.org/confluence/x/8xKhD
thanks,
Colin
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
Hi Andrew,
Stretch clusters are possible with the KRaft architecture. For example, if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1, you could put a KRaft controller node in each region. This is similar to how with ZK you'd put a ZK node in each region.
best,
Colin
On Mon, May 9, 2022, at 05:56, Andrew Otto wrote:
>> Deprecating ZK Mode and
>> Removing Zookeeper Mode
>
> I'm excited about KRaft, but quick Q. I'm researching Kafka 'stretch'
> cluster deployments, and as far as I can tell stretch clusters require
> Zookeeper to function properly, is this correct? If so, we might want to
> solve that before Deprecating ZK Mode.
>
> Thanks!
>
>
>
> On Thu, May 5, 2022 at 6:57 PM Israel Ekpo <is...@gmail.com> wrote:
>
>> Thanks Colin.
>>
>> I think we may need to update the KIP name to reflect the intent of the KIP
>> and convey everything it’s about if all the 3 action items will be covered
>> by the same KIP
>>
>> It contains three parts:
>>
>> Marking KRraft as Production Ready
>> Deprecating ZK Mode and
>> Removing Zookeeper Mode
>>
>> Should this be broken up in three separate KIPs since it will be done in
>> multiple releases?
>>
>> The current name of the KIP only conveys the first item and not the next
>> two
>>
>> It is just a thought. I will like to get your perspective
>>
>>
>>
>> On Thu, May 5, 2022 at 1:19 PM Colin McCabe <cm...@apache.org> wrote:
>>
>> > Hi all,
>> >
>> > Thanks for the comments. I agree that we should split out declaring KRaft
>> > going production for new clusters from deprecating ZK. We can do the
>> former
>> > in the next release, 3.3, and the latter in the release after that, 3.4.
>> >
>> > I also talked offline with some of the people working on upgrade from ZK
>> > and it seems like 3.4 is a more realistic target for that work. Partly
>> this
>> > is because 3.3 will be a bit earlier than I originally thought (for some
>> > reason I thought it might be October, but Ismael pointed out it's planned
>> > for August)
>> >
>> > I also agree that it will probably be useful to have a 3.5 release
>> > following the 3.4 one, which will also support ZK. Hopefully we will not
>> > need a 3.6, but we don't have to decide that now.
>> >
>> > I added a timeline section to the KIP to make this all clearer. To be
>> > clear, it is a preliminary timeline, which may change. It's difficult to
>> > fully plan out the next 1.5 years of Apache Kafka releases right now --
>> and
>> > obviously, there are things which may come up to change our plans.
>> However,
>> > I think it is still helpful to have the section to give us a feeling for
>> > the general roadmap.
>> >
>> > When you read the KIP, please consider it all speculative except for the
>> > three proposed changes at the top:
>> >
>> > 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
>> > 3.3 release.
>> > 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
>> > 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
>> > > Yes, all features supported by zk mode will be available in kraft mode
>> in
>> > > the 3.x series.
>> > >
>> > > Ismael
>> > >
>> > > On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
>> > >
>> > >> Ismael,
>> > >>
>> > >> I like the timeline. However, does this or will this also account for
>> > >> features users rely on today in Zookeeper mode being available when
>> > >> Zookeeper is dropped?
>> > >>
>> > >> That’s my main concern
>> > >>
>> > >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
>> > >>
>> > >> > Hi Colin,
>> > >> >
>> > >> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> > >> > compatibility, how about the following?
>> > >> >
>> > >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> > >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> > >> > production ready and zk mode is deprecated
>> > >> > 3. 3.5 (~April 2023): buffer release
>> > >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
>> > removed
>> > >> >
>> > >> > This would mean about 1 year from kraft being production ready to zk
>> > >> > removal and 8 months from zk deprecation to zk removal.
>> > >> >
>> > >> > If necessary (due to important bugs or security issues), we can do a
>> > >> couple
>> > >> > of additional bug fix releases in the 3.5 series after 4.0 is
>> > released.
>> > >> >
>> > >> > Thoughts?
>> > >> >
>> > >> > Ismael
>> > >> >
>> > >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> > >> >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > I've written a KIP for marking KRaft as production ready. Please
>> > take a
>> > >> > > look if you have a chance:
>> > >> > >
>> > >> > > https://cwiki.apache.org/confluence/x/8xKhD
>> > >> > >
>> > >> > > thanks,
>> > >> > > Colin
>> > >> > >
>> > >> >
>> > >> --
>> > >> Israel Ekpo
>> > >> Lead Instructor, IzzyAcademy.com
>> > >> https://www.youtube.com/c/izzyacademy
>> > >> https://izzyacademy.com/
>> > >>
>> >
>> --
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Andrew Otto <ot...@wikimedia.org>.
> Deprecating ZK Mode and
> Removing Zookeeper Mode
I'm excited about KRaft, but quick Q. I'm researching Kafka 'stretch'
cluster deployments, and as far as I can tell stretch clusters require
Zookeeper to function properly, is this correct? If so, we might want to
solve that before Deprecating ZK Mode.
Thanks!
On Thu, May 5, 2022 at 6:57 PM Israel Ekpo <is...@gmail.com> wrote:
> Thanks Colin.
>
> I think we may need to update the KIP name to reflect the intent of the KIP
> and convey everything it’s about if all the 3 action items will be covered
> by the same KIP
>
> It contains three parts:
>
> Marking KRraft as Production Ready
> Deprecating ZK Mode and
> Removing Zookeeper Mode
>
> Should this be broken up in three separate KIPs since it will be done in
> multiple releases?
>
> The current name of the KIP only conveys the first item and not the next
> two
>
> It is just a thought. I will like to get your perspective
>
>
>
> On Thu, May 5, 2022 at 1:19 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > Thanks for the comments. I agree that we should split out declaring KRaft
> > going production for new clusters from deprecating ZK. We can do the
> former
> > in the next release, 3.3, and the latter in the release after that, 3.4.
> >
> > I also talked offline with some of the people working on upgrade from ZK
> > and it seems like 3.4 is a more realistic target for that work. Partly
> this
> > is because 3.3 will be a bit earlier than I originally thought (for some
> > reason I thought it might be October, but Ismael pointed out it's planned
> > for August)
> >
> > I also agree that it will probably be useful to have a 3.5 release
> > following the 3.4 one, which will also support ZK. Hopefully we will not
> > need a 3.6, but we don't have to decide that now.
> >
> > I added a timeline section to the KIP to make this all clearer. To be
> > clear, it is a preliminary timeline, which may change. It's difficult to
> > fully plan out the next 1.5 years of Apache Kafka releases right now --
> and
> > obviously, there are things which may come up to change our plans.
> However,
> > I think it is still helpful to have the section to give us a feeling for
> > the general roadmap.
> >
> > When you read the KIP, please consider it all speculative except for the
> > three proposed changes at the top:
> >
> > 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> > 3.3 release.
> > 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> > 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
> >
> > best,
> > Colin
> >
> >
> > On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> > > Yes, all features supported by zk mode will be available in kraft mode
> in
> > > the 3.x series.
> > >
> > > Ismael
> > >
> > > On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
> > >
> > >> Ismael,
> > >>
> > >> I like the timeline. However, does this or will this also account for
> > >> features users rely on today in Zookeeper mode being available when
> > >> Zookeeper is dropped?
> > >>
> > >> That’s my main concern
> > >>
> > >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >>
> > >> > Hi Colin,
> > >> >
> > >> > Thanks for the KIP, this is exciting. Trying to balance progress and
> > >> > compatibility, how about the following?
> > >> >
> > >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> > >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> > >> > production ready and zk mode is deprecated
> > >> > 3. 3.5 (~April 2023): buffer release
> > >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> > removed
> > >> >
> > >> > This would mean about 1 year from kraft being production ready to zk
> > >> > removal and 8 months from zk deprecation to zk removal.
> > >> >
> > >> > If necessary (due to important bugs or security issues), we can do a
> > >> couple
> > >> > of additional bug fix releases in the 3.5 series after 4.0 is
> > released.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > Ismael
> > >> >
> > >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > I've written a KIP for marking KRaft as production ready. Please
> > take a
> > >> > > look if you have a chance:
> > >> > >
> > >> > > https://cwiki.apache.org/confluence/x/8xKhD
> > >> > >
> > >> > > thanks,
> > >> > > Colin
> > >> > >
> > >> >
> > >> --
> > >> Israel Ekpo
> > >> Lead Instructor, IzzyAcademy.com
> > >> https://www.youtube.com/c/izzyacademy
> > >> https://izzyacademy.com/
> > >>
> >
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
Hi José,
Yes, that matches my understanding.
I added a section on "bridge releases" to the KIP. This section basically reiterates what we discussed in KIP-500, but it's good to have anyway.
best,
Colin
On Wed, May 18, 2022, at 14:49, José Armando García Sancio wrote:
> Hi Colin,
>
> Thanks for the KIP.
>
>>The rationale for deprecating ZK in the 3.4 release is so that we can remove it in the 4.0 release. (In general, Kafka requires features to be deprecated for at least one release before they can be removed in the following major release.) During the deprecation period, ZK mode would continue to be fully supported. However, we would let people know that KRaft mode is the future.
>
> Can you also add that for existing ZK clusters the user will have to first:
> 1. Upgrade the Kafka cluster to a version between [3.3, 4.0).
> 2. Convert the ZK cluster to KRaft (pending a future KIP).
> 3. Upgrade the KRaft cluster to 4.x
>
> Does this match your understanding?
>
> --
> -José
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hi Colin,
Thanks for the KIP.
>The rationale for deprecating ZK in the 3.4 release is so that we can remove it in the 4.0 release. (In general, Kafka requires features to be deprecated for at least one release before they can be removed in the following major release.) During the deprecation period, ZK mode would continue to be fully supported. However, we would let people know that KRaft mode is the future.
Can you also add that for existing ZK clusters the user will have to first:
1. Upgrade the Kafka cluster to a version between [3.3, 4.0).
2. Convert the ZK cluster to KRaft (pending a future KIP).
3. Upgrade the KRaft cluster to 4.x
Does this match your understanding?
--
-José
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Andrew Otto <ot...@wikimedia.org>.
> Stretch clusters are possible with the KRaft architecture. For example,
if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1,
you could put a KRaft controller node in each region. This is similar to
how with ZK you'd put a ZK node in each region.
Ah ha! TIL about process.roles
<https://github.com/apache/kafka/blob/trunk/config/kraft/README.md#process-roles>.
Thank you!
On Wed, May 11, 2022 at 4:15 PM Colin McCabe <cm...@apache.org> wrote:
> On Thu, May 5, 2022, at 15:57, Israel Ekpo wrote:
> > Thanks Colin.
> >
> > I think we may need to update the KIP name to reflect the intent of the
> KIP
> > and convey everything it’s about if all the 3 action items will be
> covered
> > by the same KIP
> >
> > It contains three parts:
> >
> > Marking KRraft as Production Ready
> > Deprecating ZK Mode and
> > Removing Zookeeper Mode
> >
> > Should this be broken up in three separate KIPs since it will be done in
> > multiple releases?
>
> Hi Israel,
>
> I think these three things are closely related, and it makes sense to have
> them in a single KIP. Obviously we cannot remove ZK before deprecating it,
> or deprecate it before marking it as production-ready. So if we had 3 KIPs
> the discussion would get quite confusing since we'd constantly be
> referencing the other discussion thread(s).
>
> >
> > The current name of the KIP only conveys the first item and not the next
> > two
> >
> > It is just a thought. I will like to get your perspective
> >
>
> Names are always a hard part :) I didn't want the name to get too long.
> Maybe if others want to see a longer name I can change it -- I'm open.
>
> best,
> Colin
>
> >
> >
> > On Thu, May 5, 2022 at 1:19 PM Colin McCabe <cm...@apache.org> wrote:
> >
> >> Hi all,
> >>
> >> Thanks for the comments. I agree that we should split out declaring
> KRaft
> >> going production for new clusters from deprecating ZK. We can do the
> former
> >> in the next release, 3.3, and the latter in the release after that, 3.4.
> >>
> >> I also talked offline with some of the people working on upgrade from ZK
> >> and it seems like 3.4 is a more realistic target for that work. Partly
> this
> >> is because 3.3 will be a bit earlier than I originally thought (for some
> >> reason I thought it might be October, but Ismael pointed out it's
> planned
> >> for August)
> >>
> >> I also agree that it will probably be useful to have a 3.5 release
> >> following the 3.4 one, which will also support ZK. Hopefully we will not
> >> need a 3.6, but we don't have to decide that now.
> >>
> >> I added a timeline section to the KIP to make this all clearer. To be
> >> clear, it is a preliminary timeline, which may change. It's difficult to
> >> fully plan out the next 1.5 years of Apache Kafka releases right now --
> and
> >> obviously, there are things which may come up to change our plans.
> However,
> >> I think it is still helpful to have the section to give us a feeling for
> >> the general roadmap.
> >>
> >> When you read the KIP, please consider it all speculative except for the
> >> three proposed changes at the top:
> >>
> >> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> >> 3.3 release.
> >> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> >> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> >> > Yes, all features supported by zk mode will be available in kraft
> mode in
> >> > the 3.x series.
> >> >
> >> > Ismael
> >> >
> >> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com>
> wrote:
> >> >
> >> >> Ismael,
> >> >>
> >> >> I like the timeline. However, does this or will this also account for
> >> >> features users rely on today in Zookeeper mode being available when
> >> >> Zookeeper is dropped?
> >> >>
> >> >> That’s my main concern
> >> >>
> >> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >>
> >> >> > Hi Colin,
> >> >> >
> >> >> > Thanks for the KIP, this is exciting. Trying to balance progress
> and
> >> >> > compatibility, how about the following?
> >> >> >
> >> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> >> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> >> >> > production ready and zk mode is deprecated
> >> >> > 3. 3.5 (~April 2023): buffer release
> >> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> >> removed
> >> >> >
> >> >> > This would mean about 1 year from kraft being production ready to
> zk
> >> >> > removal and 8 months from zk deprecation to zk removal.
> >> >> >
> >> >> > If necessary (due to important bugs or security issues), we can do
> a
> >> >> couple
> >> >> > of additional bug fix releases in the 3.5 series after 4.0 is
> >> released.
> >> >> >
> >> >> > Thoughts?
> >> >> >
> >> >> > Ismael
> >> >> >
> >> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org>
> wrote:
> >> >> >
> >> >> > > Hi all,
> >> >> > >
> >> >> > > I've written a KIP for marking KRaft as production ready. Please
> >> take a
> >> >> > > look if you have a chance:
> >> >> > >
> >> >> > > https://cwiki.apache.org/confluence/x/8xKhD
> >> >> > >
> >> >> > > thanks,
> >> >> > > Colin
> >> >> > >
> >> >> >
> >> >> --
> >> >> Israel Ekpo
> >> >> Lead Instructor, IzzyAcademy.com
> >> >> https://www.youtube.com/c/izzyacademy
> >> >> https://izzyacademy.com/
> >> >>
> >>
> > --
> > Israel Ekpo
> > Lead Instructor, IzzyAcademy.com
> > https://www.youtube.com/c/izzyacademy
> > https://izzyacademy.com/
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
On Thu, May 5, 2022, at 15:57, Israel Ekpo wrote:
> Thanks Colin.
>
> I think we may need to update the KIP name to reflect the intent of the KIP
> and convey everything it’s about if all the 3 action items will be covered
> by the same KIP
>
> It contains three parts:
>
> Marking KRraft as Production Ready
> Deprecating ZK Mode and
> Removing Zookeeper Mode
>
> Should this be broken up in three separate KIPs since it will be done in
> multiple releases?
Hi Israel,
I think these three things are closely related, and it makes sense to have them in a single KIP. Obviously we cannot remove ZK before deprecating it, or deprecate it before marking it as production-ready. So if we had 3 KIPs the discussion would get quite confusing since we'd constantly be referencing the other discussion thread(s).
>
> The current name of the KIP only conveys the first item and not the next
> two
>
> It is just a thought. I will like to get your perspective
>
Names are always a hard part :) I didn't want the name to get too long. Maybe if others want to see a longer name I can change it -- I'm open.
best,
Colin
>
>
> On Thu, May 5, 2022 at 1:19 PM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi all,
>>
>> Thanks for the comments. I agree that we should split out declaring KRaft
>> going production for new clusters from deprecating ZK. We can do the former
>> in the next release, 3.3, and the latter in the release after that, 3.4.
>>
>> I also talked offline with some of the people working on upgrade from ZK
>> and it seems like 3.4 is a more realistic target for that work. Partly this
>> is because 3.3 will be a bit earlier than I originally thought (for some
>> reason I thought it might be October, but Ismael pointed out it's planned
>> for August)
>>
>> I also agree that it will probably be useful to have a 3.5 release
>> following the 3.4 one, which will also support ZK. Hopefully we will not
>> need a 3.6, but we don't have to decide that now.
>>
>> I added a timeline section to the KIP to make this all clearer. To be
>> clear, it is a preliminary timeline, which may change. It's difficult to
>> fully plan out the next 1.5 years of Apache Kafka releases right now -- and
>> obviously, there are things which may come up to change our plans. However,
>> I think it is still helpful to have the section to give us a feeling for
>> the general roadmap.
>>
>> When you read the KIP, please consider it all speculative except for the
>> three proposed changes at the top:
>>
>> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
>> 3.3 release.
>> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
>> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>>
>> best,
>> Colin
>>
>>
>> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
>> > Yes, all features supported by zk mode will be available in kraft mode in
>> > the 3.x series.
>> >
>> > Ismael
>> >
>> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
>> >
>> >> Ismael,
>> >>
>> >> I like the timeline. However, does this or will this also account for
>> >> features users rely on today in Zookeeper mode being available when
>> >> Zookeeper is dropped?
>> >>
>> >> That’s my main concern
>> >>
>> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
>> >>
>> >> > Hi Colin,
>> >> >
>> >> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> >> > compatibility, how about the following?
>> >> >
>> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> >> > production ready and zk mode is deprecated
>> >> > 3. 3.5 (~April 2023): buffer release
>> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
>> removed
>> >> >
>> >> > This would mean about 1 year from kraft being production ready to zk
>> >> > removal and 8 months from zk deprecation to zk removal.
>> >> >
>> >> > If necessary (due to important bugs or security issues), we can do a
>> >> couple
>> >> > of additional bug fix releases in the 3.5 series after 4.0 is
>> released.
>> >> >
>> >> > Thoughts?
>> >> >
>> >> > Ismael
>> >> >
>> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I've written a KIP for marking KRaft as production ready. Please
>> take a
>> >> > > look if you have a chance:
>> >> > >
>> >> > > https://cwiki.apache.org/confluence/x/8xKhD
>> >> > >
>> >> > > thanks,
>> >> > > Colin
>> >> > >
>> >> >
>> >> --
>> >> Israel Ekpo
>> >> Lead Instructor, IzzyAcademy.com
>> >> https://www.youtube.com/c/izzyacademy
>> >> https://izzyacademy.com/
>> >>
>>
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Israel Ekpo <is...@gmail.com>.
Thanks Colin.
I think we may need to update the KIP name to reflect the intent of the KIP
and convey everything it’s about if all the 3 action items will be covered
by the same KIP
It contains three parts:
Marking KRraft as Production Ready
Deprecating ZK Mode and
Removing Zookeeper Mode
Should this be broken up in three separate KIPs since it will be done in
multiple releases?
The current name of the KIP only conveys the first item and not the next
two
It is just a thought. I will like to get your perspective
On Thu, May 5, 2022 at 1:19 PM Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> Thanks for the comments. I agree that we should split out declaring KRaft
> going production for new clusters from deprecating ZK. We can do the former
> in the next release, 3.3, and the latter in the release after that, 3.4.
>
> I also talked offline with some of the people working on upgrade from ZK
> and it seems like 3.4 is a more realistic target for that work. Partly this
> is because 3.3 will be a bit earlier than I originally thought (for some
> reason I thought it might be October, but Ismael pointed out it's planned
> for August)
>
> I also agree that it will probably be useful to have a 3.5 release
> following the 3.4 one, which will also support ZK. Hopefully we will not
> need a 3.6, but we don't have to decide that now.
>
> I added a timeline section to the KIP to make this all clearer. To be
> clear, it is a preliminary timeline, which may change. It's difficult to
> fully plan out the next 1.5 years of Apache Kafka releases right now -- and
> obviously, there are things which may come up to change our plans. However,
> I think it is still helpful to have the section to give us a feeling for
> the general roadmap.
>
> When you read the KIP, please consider it all speculative except for the
> three proposed changes at the top:
>
> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> 3.3 release.
> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>
> best,
> Colin
>
>
> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> > Yes, all features supported by zk mode will be available in kraft mode in
> > the 3.x series.
> >
> > Ismael
> >
> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
> >
> >> Ismael,
> >>
> >> I like the timeline. However, does this or will this also account for
> >> features users rely on today in Zookeeper mode being available when
> >> Zookeeper is dropped?
> >>
> >> That’s my main concern
> >>
> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
> >>
> >> > Hi Colin,
> >> >
> >> > Thanks for the KIP, this is exciting. Trying to balance progress and
> >> > compatibility, how about the following?
> >> >
> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> >> > production ready and zk mode is deprecated
> >> > 3. 3.5 (~April 2023): buffer release
> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> removed
> >> >
> >> > This would mean about 1 year from kraft being production ready to zk
> >> > removal and 8 months from zk deprecation to zk removal.
> >> >
> >> > If necessary (due to important bugs or security issues), we can do a
> >> couple
> >> > of additional bug fix releases in the 3.5 series after 4.0 is
> released.
> >> >
> >> > Thoughts?
> >> >
> >> > Ismael
> >> >
> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I've written a KIP for marking KRaft as production ready. Please
> take a
> >> > > look if you have a chance:
> >> > >
> >> > > https://cwiki.apache.org/confluence/x/8xKhD
> >> > >
> >> > > thanks,
> >> > > Colin
> >> > >
> >> >
> >> --
> >> Israel Ekpo
> >> Lead Instructor, IzzyAcademy.com
> >> https://www.youtube.com/c/izzyacademy
> >> https://izzyacademy.com/
> >>
>
--
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
Hi all,
Thanks for the comments. I agree that we should split out declaring KRaft going production for new clusters from deprecating ZK. We can do the former in the next release, 3.3, and the latter in the release after that, 3.4.
I also talked offline with some of the people working on upgrade from ZK and it seems like 3.4 is a more realistic target for that work. Partly this is because 3.3 will be a bit earlier than I originally thought (for some reason I thought it might be October, but Ismael pointed out it's planned for August)
I also agree that it will probably be useful to have a 3.5 release following the 3.4 one, which will also support ZK. Hopefully we will not need a 3.6, but we don't have to decide that now.
I added a timeline section to the KIP to make this all clearer. To be clear, it is a preliminary timeline, which may change. It's difficult to fully plan out the next 1.5 years of Apache Kafka releases right now -- and obviously, there are things which may come up to change our plans. However, I think it is still helpful to have the section to give us a feeling for the general roadmap.
When you read the KIP, please consider it all speculative except for the three proposed changes at the top:
1. Mark KRaft as production-ready for new clusters in the upcoming Kafka 3.3 release.
2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
best,
Colin
On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> Yes, all features supported by zk mode will be available in kraft mode in
> the 3.x series.
>
> Ismael
>
> On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
>
>> Ismael,
>>
>> I like the timeline. However, does this or will this also account for
>> features users rely on today in Zookeeper mode being available when
>> Zookeeper is dropped?
>>
>> That’s my main concern
>>
>> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
>>
>> > Hi Colin,
>> >
>> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> > compatibility, how about the following?
>> >
>> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> > production ready and zk mode is deprecated
>> > 3. 3.5 (~April 2023): buffer release
>> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
>> >
>> > This would mean about 1 year from kraft being production ready to zk
>> > removal and 8 months from zk deprecation to zk removal.
>> >
>> > If necessary (due to important bugs or security issues), we can do a
>> couple
>> > of additional bug fix releases in the 3.5 series after 4.0 is released.
>> >
>> > Thoughts?
>> >
>> > Ismael
>> >
>> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I've written a KIP for marking KRaft as production ready. Please take a
>> > > look if you have a chance:
>> > >
>> > > https://cwiki.apache.org/confluence/x/8xKhD
>> > >
>> > > thanks,
>> > > Colin
>> > >
>> >
>> --
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Ismael Juma <is...@juma.me.uk>.
Yes, all features supported by zk mode will be available in kraft mode in
the 3.x series.
Ismael
On Wed, May 4, 2022, 5:28 PM Israel Ekpo <is...@gmail.com> wrote:
> Ismael,
>
> I like the timeline. However, does this or will this also account for
> features users rely on today in Zookeeper mode being available when
> Zookeeper is dropped?
>
> That’s my main concern
>
> On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Colin,
> >
> > Thanks for the KIP, this is exciting. Trying to balance progress and
> > compatibility, how about the following?
> >
> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> > production ready and zk mode is deprecated
> > 3. 3.5 (~April 2023): buffer release
> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
> >
> > This would mean about 1 year from kraft being production ready to zk
> > removal and 8 months from zk deprecation to zk removal.
> >
> > If necessary (due to important bugs or security issues), we can do a
> couple
> > of additional bug fix releases in the 3.5 series after 4.0 is released.
> >
> > Thoughts?
> >
> > Ismael
> >
> > On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > I've written a KIP for marking KRaft as production ready. Please take a
> > > look if you have a chance:
> > >
> > > https://cwiki.apache.org/confluence/x/8xKhD
> > >
> > > thanks,
> > > Colin
> > >
> >
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Israel Ekpo <is...@gmail.com>.
Ismael,
I like the timeline. However, does this or will this also account for
features users rely on today in Zookeeper mode being available when
Zookeeper is dropped?
That’s my main concern
On Wed, May 4, 2022 at 8:12 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Colin,
>
> Thanks for the KIP, this is exciting. Trying to balance progress and
> compatibility, how about the following?
>
> 1. 3.3 (~August 2022): kraft is production ready for new clusters
> 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> production ready and zk mode is deprecated
> 3. 3.5 (~April 2023): buffer release
> 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
>
> This would mean about 1 year from kraft being production ready to zk
> removal and 8 months from zk deprecation to zk removal.
>
> If necessary (due to important bugs or security issues), we can do a couple
> of additional bug fix releases in the 3.5 series after 4.0 is released.
>
> Thoughts?
>
> Ismael
>
> On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > I've written a KIP for marking KRaft as production ready. Please take a
> > look if you have a chance:
> >
> > https://cwiki.apache.org/confluence/x/8xKhD
> >
> > thanks,
> > Colin
> >
>
--
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Colin,
Thanks for the KIP, this is exciting. Trying to balance progress and
compatibility, how about the following?
1. 3.3 (~August 2022): kraft is production ready for new clusters
2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
production ready and zk mode is deprecated
3. 3.5 (~April 2023): buffer release
4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
This would mean about 1 year from kraft being production ready to zk
removal and 8 months from zk deprecation to zk removal.
If necessary (due to important bugs or security issues), we can do a couple
of additional bug fix releases in the 3.5 series after 4.0 is released.
Thoughts?
Ismael
On Tue, May 3, 2022, 6:03 PM Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Christo <ch...@yahoo.com.INVALID>.
Hello Colin,
Thank you for the swift and informative reply! I will have a look at the suggested tickets and pick up one of the free ones.
Best,
Christo
On Sunday, 19 June 2022, 22:46:41 BST, Colin McCabe <cm...@apache.org> wrote:
Hi Christo NUKK,
We have been trying to label KRaft issues on JIRA with "kip-500". So you should be able to do a search like this for them:
https://issues.apache.org/jira/browse/KAFKA-13912?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20kip-500%20AND%20statusCategory%20!%3D%20Done
In JQL terms, that's "project = KAFKA AND labels = kip-500 AND statusCategory != Done"
The dynamic configuration piece is a bit tricky. You might consider starting with something like KAFKA-13650, KAFKA-13774, or KAFKA-13620.
best,
Colin
On Sat, Jun 18, 2022, at 23:21, Christo NUKK wrote:
> Hello!
> I went through JIRA, GitHub and the mailing list, but I was not able to
> find a list of developer tasks to be picked up for what is left to
> achieve feature parity with Zookeeper mode. I will have spare bandwidth
> in the next month and would like to contribute. I have touched
> configurations before so I would want to pick up Modifying certain
> dynamic configurations on the standalone KRaft controller if possible.
> Best,Christo
> On Wednesday, 4 May 2022, 02:03:47 BST, Colin McCabe
> <cm...@apache.org> wrote:
>
> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
Hi Christo NUKK,
We have been trying to label KRaft issues on JIRA with "kip-500". So you should be able to do a search like this for them:
https://issues.apache.org/jira/browse/KAFKA-13912?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20kip-500%20AND%20statusCategory%20!%3D%20Done
In JQL terms, that's "project = KAFKA AND labels = kip-500 AND statusCategory != Done"
The dynamic configuration piece is a bit tricky. You might consider starting with something like KAFKA-13650, KAFKA-13774, or KAFKA-13620.
best,
Colin
On Sat, Jun 18, 2022, at 23:21, Christo NUKK wrote:
> Hello!
> I went through JIRA, GitHub and the mailing list, but I was not able to
> find a list of developer tasks to be picked up for what is left to
> achieve feature parity with Zookeeper mode. I will have spare bandwidth
> in the next month and would like to contribute. I have touched
> configurations before so I would want to pick up Modifying certain
> dynamic configurations on the standalone KRaft controller if possible.
> Best,Christo
> On Wednesday, 4 May 2022, 02:03:47 BST, Colin McCabe
> <cm...@apache.org> wrote:
>
> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Christo NUKK <ch...@yahoo.com.INVALID>.
Hello!
I went through JIRA, GitHub and the mailing list, but I was not able to find a list of developer tasks to be picked up for what is left to achieve feature parity with Zookeeper mode. I will have spare bandwidth in the next month and would like to contribute. I have touched configurations before so I would want to pick up Modifying certain dynamic configurations on the standalone KRaft controller if possible.
Best,Christo
On Wednesday, 4 May 2022, 02:03:47 BST, Colin McCabe <cm...@apache.org> wrote:
Hi all,
I've written a KIP for marking KRaft as production ready. Please take a look if you have a chance:
https://cwiki.apache.org/confluence/x/8xKhD
thanks,
Colin
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Israel Ekpo <is...@gmail.com>.
Thanks Colin for the update and clarification and the additional details
you added.
I second Igor's thoughts that we should not deprecate ZK before having
feature parity.
However, I think marking KRaft mode as production-ready is something that
can be done independent of the status of feature parity (between ZK-mode
and KRaft-mode).
Maybe we can mark KRaft as production ready for now (in 3.3) then work on
getting to feature parity in 3.4 and once feature parity is attained, then
we deprecate ZK-mode in 3.4 and remove it in the next release after 3.4.
That would be fair from my perspective.
It is a bit hard to ask users to jump off the boat when the dock is still
far away and they are unable to swim that far.
Some of the missing features are still in the critical path for some users
and we should include them in the project before the ZK-mode deprecation or
removal.
These are my initial thoughts about this and I believe users may not feel
supported if the cord is just ripped off with features still missing and
Zookeeper is gone.
That could send a wrong message to the community of current and prospective
users.
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/
On Wed, May 4, 2022 at 5:58 AM Igor Soarez <i...@soarez.me> wrote:
> Hi Colin,
>
> It's very exciting to see KRaft ready for production!
>
> I see the value of marking it ready for production even with the current
> missing features.
>
> However, I'm worried about marking ZK mode as deprecated before these
> missing features are ready. It might be hard to estimate right now how much
> work is actually necessary to close that gap, particularly the upgrade from
> ZK. Deprecating ZK before providing users with an upgrade path might be
> sending the wrong message - when something is deprecated we want users to
> move away from it, so that should be possible. It might well be more than a
> few months to close the gap.
>
> Best,
>
> --
> Igor
>
> On Wed, May 4, 2022, at 7:22 AM, Colin McCabe wrote:
> > On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
> >>
> >> To be clear, the proposal here is to have the bridge release be 3.4 and
> >> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not
> >> requirement) if we can't finish everything in time after 3.4. So that
> >> would be two more releases with ZK, or dropping ZK a little bit less
> >> than a year from now.
> >>
> >
> > Sigh. This should read "To be clear, the proposal here is to have the
> > bridge release be 3.3 and then move on to a ZK-free 4.0. With a 3.4
> > release as an option (but not requirement) if we can't finish
> > everything in time after 3.3." So much for being clear, I guess :)
> >
> > cheers,
> > Colin
> >
> >
> >> best,
> >> Colin
> >>
> >>>
> >>> [1]
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> >>> [2] https://kafka.apache.org/downloads
> >>>
> >>>
> >>> Israel Ekpo
> >>> Lead Instructor, IzzyAcademy.com
> >>> https://www.youtube.com/c/izzyacademy
> >>> https://izzyacademy.com/
> >>>
> >>>
> >>> On Tue, May 3, 2022 at 10:32 PM Luke Chen <sh...@gmail.com> wrote:
> >>>
> >>>> Hi Colin,
> >>>>
> >>>> So exciting to see the KIP to mark the Kraft production ready!
> >>>>
> >>>> Just one comment: We should make sure the period between ZK
> deprecation
> >>>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> >>>> Do we have any expectation for the deprecation period?
> >>>> After all, this is not a small feature change, and users need more
> time to
> >>>> do the migration.
> >>>> But to be honest, I don't know how long is long enough.
> >>>> Maybe at least half a year? Or more? Thoughts?
> >>>>
> >>>>
> >>>> Thank you.
> >>>> Luke
> >>>>
> >>>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org>
> wrote:
> >>>>
> >>>> > Hi all,
> >>>> >
> >>>> > I've written a KIP for marking KRaft as production ready. Please
> take a
> >>>> > look if you have a chance:
> >>>> >
> >>>> > https://cwiki.apache.org/confluence/x/8xKhD
> >>>> >
> >>>> > thanks,
> >>>> > Colin
> >>>> >
> >>>>
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Igor Soarez <i...@soarez.me>.
Hi Colin,
It's very exciting to see KRaft ready for production!
I see the value of marking it ready for production even with the current missing features.
However, I'm worried about marking ZK mode as deprecated before these missing features are ready. It might be hard to estimate right now how much work is actually necessary to close that gap, particularly the upgrade from ZK. Deprecating ZK before providing users with an upgrade path might be sending the wrong message - when something is deprecated we want users to move away from it, so that should be possible. It might well be more than a few months to close the gap.
Best,
--
Igor
On Wed, May 4, 2022, at 7:22 AM, Colin McCabe wrote:
> On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
>>
>> To be clear, the proposal here is to have the bridge release be 3.4 and
>> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not
>> requirement) if we can't finish everything in time after 3.4. So that
>> would be two more releases with ZK, or dropping ZK a little bit less
>> than a year from now.
>>
>
> Sigh. This should read "To be clear, the proposal here is to have the
> bridge release be 3.3 and then move on to a ZK-free 4.0. With a 3.4
> release as an option (but not requirement) if we can't finish
> everything in time after 3.3." So much for being clear, I guess :)
>
> cheers,
> Colin
>
>
>> best,
>> Colin
>>
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>>> [2] https://kafka.apache.org/downloads
>>>
>>>
>>> Israel Ekpo
>>> Lead Instructor, IzzyAcademy.com
>>> https://www.youtube.com/c/izzyacademy
>>> https://izzyacademy.com/
>>>
>>>
>>> On Tue, May 3, 2022 at 10:32 PM Luke Chen <sh...@gmail.com> wrote:
>>>
>>>> Hi Colin,
>>>>
>>>> So exciting to see the KIP to mark the Kraft production ready!
>>>>
>>>> Just one comment: We should make sure the period between ZK deprecation
>>>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>>>> Do we have any expectation for the deprecation period?
>>>> After all, this is not a small feature change, and users need more time to
>>>> do the migration.
>>>> But to be honest, I don't know how long is long enough.
>>>> Maybe at least half a year? Or more? Thoughts?
>>>>
>>>>
>>>> Thank you.
>>>> Luke
>>>>
>>>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
>>>>
>>>> > Hi all,
>>>> >
>>>> > I've written a KIP for marking KRaft as production ready. Please take a
>>>> > look if you have a chance:
>>>> >
>>>> > https://cwiki.apache.org/confluence/x/8xKhD
>>>> >
>>>> > thanks,
>>>> > Colin
>>>> >
>>>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
>
> To be clear, the proposal here is to have the bridge release be 3.4 and
> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not
> requirement) if we can't finish everything in time after 3.4. So that
> would be two more releases with ZK, or dropping ZK a little bit less
> than a year from now.
>
Sigh. This should read "To be clear, the proposal here is to have the bridge release be 3.3 and then move on to a ZK-free 4.0. With a 3.4 release as an option (but not requirement) if we can't finish everything in time after 3.3." So much for being clear, I guess :)
cheers,
Colin
> best,
> Colin
>
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>> [2] https://kafka.apache.org/downloads
>>
>>
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>
>>
>> On Tue, May 3, 2022 at 10:32 PM Luke Chen <sh...@gmail.com> wrote:
>>
>>> Hi Colin,
>>>
>>> So exciting to see the KIP to mark the Kraft production ready!
>>>
>>> Just one comment: We should make sure the period between ZK deprecation
>>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>>> Do we have any expectation for the deprecation period?
>>> After all, this is not a small feature change, and users need more time to
>>> do the migration.
>>> But to be honest, I don't know how long is long enough.
>>> Maybe at least half a year? Or more? Thoughts?
>>>
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
>>>
>>> > Hi all,
>>> >
>>> > I've written a KIP for marking KRaft as production ready. Please take a
>>> > look if you have a chance:
>>> >
>>> > https://cwiki.apache.org/confluence/x/8xKhD
>>> >
>>> > thanks,
>>> > Colin
>>> >
>>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
On Tue, May 3, 2022, at 20:48, Israel Ekpo wrote:
> Hi Colin,
>
> Thanks for this KIP. I am very excited to see the proposal.
>
> A lot in the community have been asking when KRaft mode will be ready and I
> think this KIP will provide a lot of the answers and guidance desperately
> needed.
>
> Thanks for working on it.
>
> I also have some of the questions from Luke regarding the duration of the
> deprecation period and the release where ZK will be completely dropped.
>
> As per our time-based release plan [1], the release should be every 4
> months, but it has been approximately 5 months between releases from the
> downloads page [2]
>
> So, right now we are about to push out 3.2.0 and 3.4.0 is approximately 10
> months from now based on the schedule/plan.
>
Hi Israel,
Sorry, I meant to write "the next release, 3.3" not "the next release, 3.4." I fixed the KIP text now.
As you noted, 3.3 should be about 4 or 5 months from now.
> I wonder if this will be enough time to complete all the features needed to
> bring KRaft mode to parity with Legacy/Zookeeper mode.
>
> The missing features you have highlighted and any other critical features
> not working in KRaft mode should be implemented fully before marking KRaft
> mode as production ready and deprecating Legacy mode.
>
As I wrote in "potential objections and responses," I don't think we need full feature parity before marking KRaft mode as production ready. It's customary for Kafka to bring new features to production incrementally.
>
> The main question here is would we have Kafka versions 3.6.0, 3.7.0, 3.8.0
> and 3.9.0 because this will add approximately 20 additional months after
> 3.4.0 which should be enough time to have ZK-mode deprecated and tested
> thoroughly before removal completely. I feel we should not mark ZK mode as
> deprecated until all the features in KRaft mode are at parity with ZK-mode.
>
> If we are going to have 20 months after 3.4 to work with a deprecated
> ZK-mode, then it could be possible to even drop ZK fully before 4.0
>
> Let's move the boat closer to the dock before we jump out of the boat. It
> will be very stressful if we jump out of the boat and have to swim several
> miles to shore.
>
That is a lot of Kafka versions! I don't think anyone wants 3.x to go on that long, or the ZooKeeper metadata implementation either. Ultimately I believe such a very long multi-year transition period would make the code quality and project stability worse, since we'd have to split our efforts across the two implementations.
To be clear, the proposal here is to have the bridge release be 3.4 and then move on to a ZK-free 4.0. With a 3.5 release as an option (but not requirement) if we can't finish everything in time after 3.4. So that would be two more releases with ZK, or dropping ZK a little bit less than a year from now.
best,
Colin
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> [2] https://kafka.apache.org/downloads
>
>
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>
>
> On Tue, May 3, 2022 at 10:32 PM Luke Chen <sh...@gmail.com> wrote:
>
>> Hi Colin,
>>
>> So exciting to see the KIP to mark the Kraft production ready!
>>
>> Just one comment: We should make sure the period between ZK deprecation
>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>> Do we have any expectation for the deprecation period?
>> After all, this is not a small feature change, and users need more time to
>> do the migration.
>> But to be honest, I don't know how long is long enough.
>> Maybe at least half a year? Or more? Thoughts?
>>
>>
>> Thank you.
>> Luke
>>
>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
>>
>> > Hi all,
>> >
>> > I've written a KIP for marking KRaft as production ready. Please take a
>> > look if you have a chance:
>> >
>> > https://cwiki.apache.org/confluence/x/8xKhD
>> >
>> > thanks,
>> > Colin
>> >
>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Israel Ekpo <is...@gmail.com>.
Hi Colin,
Thanks for this KIP. I am very excited to see the proposal.
A lot in the community have been asking when KRaft mode will be ready and I
think this KIP will provide a lot of the answers and guidance desperately
needed.
Thanks for working on it.
I also have some of the questions from Luke regarding the duration of the
deprecation period and the release where ZK will be completely dropped.
As per our time-based release plan [1], the release should be every 4
months, but it has been approximately 5 months between releases from the
downloads page [2]
So, right now we are about to push out 3.2.0 and 3.4.0 is approximately 10
months from now based on the schedule/plan.
I wonder if this will be enough time to complete all the features needed to
bring KRaft mode to parity with Legacy/Zookeeper mode.
The missing features you have highlighted and any other critical features
not working in KRaft mode should be implemented fully before marking KRaft
mode as production ready and deprecating Legacy mode.
The main question here is would we have Kafka versions 3.6.0, 3.7.0, 3.8.0
and 3.9.0 because this will add approximately 20 additional months after
3.4.0 which should be enough time to have ZK-mode deprecated and tested
thoroughly before removal completely. I feel we should not mark ZK mode as
deprecated until all the features in KRaft mode are at parity with ZK-mode.
If we are going to have 20 months after 3.4 to work with a deprecated
ZK-mode, then it could be possible to even drop ZK fully before 4.0
Let's move the boat closer to the dock before we jump out of the boat. It
will be very stressful if we jump out of the boat and have to swim several
miles to shore.
[1]
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
[2] https://kafka.apache.org/downloads
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/
On Tue, May 3, 2022 at 10:32 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Colin,
>
> So exciting to see the KIP to mark the Kraft production ready!
>
> Just one comment: We should make sure the period between ZK deprecation
> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> Do we have any expectation for the deprecation period?
> After all, this is not a small feature change, and users need more time to
> do the migration.
> But to be honest, I don't know how long is long enough.
> Maybe at least half a year? Or more? Thoughts?
>
>
> Thank you.
> Luke
>
> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi all,
> >
> > I've written a KIP for marking KRaft as production ready. Please take a
> > look if you have a chance:
> >
> > https://cwiki.apache.org/confluence/x/8xKhD
> >
> > thanks,
> > Colin
> >
>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Colin McCabe <cm...@apache.org>.
On Tue, May 3, 2022, at 19:32, Luke Chen wrote:
> Hi Colin,
>
> So exciting to see the KIP to mark the Kraft production ready!
>
> Just one comment: We should make sure the period between ZK deprecation
> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> Do we have any expectation for the deprecation period?
> After all, this is not a small feature change, and users need more time to
> do the migration.
> But to be honest, I don't know how long is long enough.
> Maybe at least half a year? Or more? Thoughts?
>
Hi Luke,
Like we discussed below, in chronological terms a Kafka release is about 4 or 5 months. So if we have 3.3, followed by a 4.0 where we drop ZK, that would be about 8 to 10 months. If we need more time we can add a 3.4 which will add another 4-5 months in between.
best,
Colin
>
> Thank you.
> Luke
>
> On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
>
>> Hi all,
>>
>> I've written a KIP for marking KRaft as production ready. Please take a
>> look if you have a chance:
>>
>> https://cwiki.apache.org/confluence/x/8xKhD
>>
>> thanks,
>> Colin
>>
Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready
Posted by Luke Chen <sh...@gmail.com>.
Hi Colin,
So exciting to see the KIP to mark the Kraft production ready!
Just one comment: We should make sure the period between ZK deprecation
(i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
Do we have any expectation for the deprecation period?
After all, this is not a small feature change, and users need more time to
do the migration.
But to be honest, I don't know how long is long enough.
Maybe at least half a year? Or more? Thoughts?
Thank you.
Luke
On Wed, May 4, 2022 at 9:10 AM Colin McCabe <cm...@apache.org> wrote:
> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
>