You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Ryan Dielhenn <rd...@confluent.io.INVALID> on 2021/08/03 19:13:38 UTC
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Hi Konstantine,
I would like to report another bug in KRaft.
The ConfigHandler that processes dynamic broker config deltas in KRaft
expects that the default resource name for dynamic broker configs is the
old default entity name used in ZK: "<default>". Since dynamic default
broker configs are persisted as empty string in the quorum instead of
"<default>", the brokers are not updating the their default configuration
when they see empty string as a resource name in the config delta and are
throwing a NumberFormatException when they try to parse the resource name
to process it as a per-broker configuration.
I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
I also have a PR to fix this: https://github.com/apache/kafka/pull/11168
I think that this should be a blocker for 3.0 because dynamic default
broker configs will not be usable in KRaft otherwise.
Best,
Ryan Dielhenn
On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
> Thanks Ryan,
>
> Approved. Seems also like a low risk fix.
> With that opportunity, let's make sure there are no other configs that
> would need a similar validation.
>
> Konstantine
>
> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> <rd...@confluent.io.invalid> wrote:
>
> > Hey Konstantine,
> >
> > Thanks for the question. If these configs are not validated the user's
> > experience will be affected and upgrades from 3.0 will be harder.
> >
> > Best,
> > Ryan Dielhenn
> >
> > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Thanks for reporting this issue Ryan.
> > >
> > > I believe what you mention corresponds to the ticket you created here:
> > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> > >
> > > What happens if the configurations are present but the broker doesn't
> > fail
> > > at startup when configured to run in KRaft mode?
> > > Asking to see if we have any workarounds in our availability.
> > >
> > > Thanks,
> > > Konstantine
> > >
> > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> > > <rd...@confluent.io.invalid> wrote:
> > >
> > > > Hi,
> > > >
> > > > Disregard log.clean.policy being included in this blocker.
> > > >
> > > > Best,
> > > > Ryan Dielhenn
> > > >
> > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> rdielhenn@confluent.io>
> > > > wrote:
> > > >
> > > > > Hey Konstantine,
> > > > >
> > > > > I'd like to report another bug in KRaft.
> > > > >
> > > > > log.cleanup.policy, alter.config.policy.class.name, and
> > > > > create.topic.policy.class.name are all unsupported by KRaft but
> > KRaft
> > > > > servers allow them to be configured. I believe this should be
> > > considered
> > > > a
> > > > > blocker and that KRaft servers should fail startup if any of these
> > are
> > > > > configured. I do not have a PR yet but will soon.
> > > > >
> > > > > On another note, I have a PR for the dynamic broker configuration
> fix
> > > > > here: https://github.com/apache/kafka/pull/11141
> > > > >
> > > > > Best,
> > > > > Ryan Dielhenn
> > > > >
> > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > > > > <ko...@confluent.io.invalid> wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> Please find below the updated release plan for the Apache Kafka
> > 3.0.0
> > > > >> release.
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > > >>
> > > > >> New suggested dates for the release are as follows:
> > > > >>
> > > > >> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > > >> Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > > > >> Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > > > >>
> > > > >> At least two weeks of stabilization will follow Code Freeze.
> > > > >>
> > > > >> The release plan is up to date and currently includes all the
> > approved
> > > > >> KIPs
> > > > >> that are targeting 3.0.0.
> > > > >>
> > > > >> Please let me know if you have any objections with the recent
> > > extension
> > > > of
> > > > >> Feature Freeze and Code Freeze or any other concerns.
> > > > >>
> > > > >> Regards,
> > > > >> Konstantine
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Konstantine Karantasis <kk...@apache.org>.
Thanks for the brief discussion Matthias and Israel.
Since there's no option but to merge a change, at this late stage in the
release process I'd agree with Matthias's suggestion to move forward with a
partial reversion of changes, which seems to be the safest of the two
options. I'll port the changes to the release plan and the blogpost I'm
preparing.
Hi Ron. Thanks for reporting the new issue. Indeed, we are quite picky
about what constitutes a blocker at this point. I was about to start
preparing the first RC earlier today.
The issue you report seems to meet our bar. Given that it's low risk, I'm
in favor of including it if we can merge the PR you prepared some time
soon.
Thank you all,
Konstantine
On Thu, Aug 19, 2021 at 6:09 PM Ron Dagostino <rn...@gmail.com> wrote:
> Hi Konstantine. A potential 3.0 blocker was discovered today,
> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13219. The
> BrokerState metric is not working for KRaft clusters -- it always indicates
> that the broker is in the "not running" state. A PR is available at
> https://github.com/apache/kafka/pull/11239. The risk is low, and there is
> no workaround, but I also understand the bar for a blocker is pretty high
> at this point.
>
> Ron
>
> On Thu, Aug 19, 2021 at 5:54 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Konstantine,
> >
> > After a brief chat with Sophie, I just updated the name for KIP-633 to be
> > more descriptive of what is actually happening in the implementation
> >
> > It is changed on the KIP page and the JIRA task title is also updated to
> be
> > more descriptive
> > https://cwiki.apache.org/confluence/x/Ho2NCg
> > https://issues.apache.org/jira/browse/KAFKA-8613
> >
> > However, I could not update the page for the 3.0 release plan. I do not
> > have the permissions for that update.
> > https://cwiki.apache.org/confluence/x/woONCg
> >
> > When you have a moment, please could you update the release plan for 3.0
> to
> > reflect the update name for the KIP?
> >
> > Thank you very much
> >
> > Sincerely,
> > Israel
> >
> >
> > On Thu, Aug 19, 2021 at 5:50 PM Israel Ekpo <is...@gmail.com>
> wrote:
> >
> > > Hi Matthias, it is possible to still deprecate JoinWindows.of(size)
> even
> > > though the new join semantics is disabled.
> > >
> > > I just pre-recorded a talk for Kafka Summit Americas where I am
> > > recommending a switch to the new APIs instead of the deprecated one
> > > starting from 3.0
> > >
> > > I would love to be involved in the discussion for the fix so that we
> try
> > > to honor the expectations of the KIP as much as possible.
> > >
> > > So far, it looks like the PR will still honor the expectations of the
> KIP
> > >
> > > https://github.com/apache/kafka/pull/11235
> > >
> > > Thanks for sharing this to create awareness.
> > >
> > > Sincerely,
> > > Israel
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org>
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> we discovered a potential blocker for 3.0.0 today:
> > >> https://issues.apache.org/jira/browse/KAFKA-13216
> > >>
> > >> We are still evaluating a potential fix. If we cannot fix it quickly,
> > >> the fall-back would be to partially roll-back KIP-633, to disable the
> > >> new join semantics such that people cannot hit this bug.
> > >>
> > >> Thoughts?
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >> On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
> > >> > Hi Rajini.
> > >> >
> > >> > Approved, given its low risk and the lack of convenient workarounds.
> > >> >
> > >> > Konstantine
> > >> >
> > >> >
> > >> > On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <
> > >> rajinisivaram@gmail.com>
> > >> > wrote:
> > >> >
> > >> >> Hi Konstantine,
> > >> >>
> > >> >> We found an issue with replication with IBP 2.7:
> > >> >> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is
> small
> > >> and
> > >> >> low
> > >> >> risk and has been merged to trunk. Can we include this in 3.0
> branch
> > >> since
> > >> >> it can result in IllegalStateException during replication?
> > >> >>
> > >> >> Thank you,
> > >> >>
> > >> >> Rajini
> > >> >>
> > >> >>
> > >> >> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
> > >> >> kkarantasis@apache.org> wrote:
> > >> >>
> > >> >>> Jose, thanks for the heads up on the 3 new blocker candidates.
> > >> >>>
> > >> >>> I read the tickets and they have clear descriptions and
> > implementation
> > >> >>> details.
> > >> >>> However, at this stage to be able to make a call and approve new
> > >> blockers
> > >> >>> I'd appreciate it if we could get some insight regarding the risk
> > and
> > >> the
> > >> >>> necessity of a fix. A rough ETA would also be helpful.
> > >> >>>
> > >> >>> Having said that, based on the descriptions and the existence of a
> > few
> > >> >>> other blockers, I'm tentatively approving KAFKA-13161,
> KAFKA-13165,
> > >> and
> > >> >>> KAFKA-13168 and we might have to make a new assessment if these
> are
> > >> the
> > >> >>> only blockers in the next few days or if we notice a regression
> > during
> > >> >>> testing.
> > >> >>>
> > >> >>> Konstantine
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
> > >> >>> kkarantasis@apache.org> wrote:
> > >> >>>
> > >> >>>>
> > >> >>>> Thanks for reporting this new issue Ryan,
> > >> >>>>
> > >> >>>> It's important and this issue seems to have clearly regressed
> > dynamic
> > >> >>>> default configs in the 3.0 branch.
> > >> >>>> So, it's approved.
> > >> >>>>
> > >> >>>> Konstantine
> > >> >>>>
> > >> >>>>
> > >> >>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> > >> >>>> <js...@confluent.io.invalid> wrote:
> > >> >>>>
> > >> >>>>> Hey all,
> > >> >>>>>
> > >> >>>>> For the KIP-500 work for 3.0 we would like to propose the
> > following
> > >> >>>>> Jiras as blockers:
> > >> >>>>>
> > >> >>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> > >> >>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> > >> >>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
> > >> >>>>>
> > >> >>>>> The description for each Jira should have more details.
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> -Jose
> > >> >>>>>
> > >> >>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> > >> >>>>> <rd...@confluent.io.invalid> wrote:
> > >> >>>>>>
> > >> >>>>>> Hi Konstantine,
> > >> >>>>>>
> > >> >>>>>> I would like to report another bug in KRaft.
> > >> >>>>>>
> > >> >>>>>> The ConfigHandler that processes dynamic broker config deltas
> in
> > >> >> KRaft
> > >> >>>>>> expects that the default resource name for dynamic broker
> configs
> > >> is
> > >> >>> the
> > >> >>>>>> old default entity name used in ZK: "<default>". Since dynamic
> > >> >> default
> > >> >>>>>> broker configs are persisted as empty string in the quorum
> > instead
> > >> >> of
> > >> >>>>>> "<default>", the brokers are not updating the their default
> > >> >>>>> configuration
> > >> >>>>>> when they see empty string as a resource name in the config
> delta
> > >> >> and
> > >> >>>>> are
> > >> >>>>>> throwing a NumberFormatException when they try to parse the
> > >> resource
> > >> >>>>> name
> > >> >>>>>> to process it as a per-broker configuration.
> > >> >>>>>>
> > >> >>>>>> I filed a JIRA:
> > https://issues.apache.org/jira/browse/KAFKA-13160
> > >> >>>>>>
> > >> >>>>>> I also have a PR to fix this:
> > >> >>>>> https://github.com/apache/kafka/pull/11168
> > >> >>>>>>
> > >> >>>>>> I think that this should be a blocker for 3.0 because dynamic
> > >> >> default
> > >> >>>>>> broker configs will not be usable in KRaft otherwise.
> > >> >>>>>>
> > >> >>>>>> Best,
> > >> >>>>>> Ryan Dielhenn
> > >> >>>>>>
> > >> >>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> > >> >>>>>> kkarantasis@apache.org> wrote:
> > >> >>>>>>
> > >> >>>>>>> Thanks Ryan,
> > >> >>>>>>>
> > >> >>>>>>> Approved. Seems also like a low risk fix.
> > >> >>>>>>> With that opportunity, let's make sure there are no other
> > configs
> > >> >>> that
> > >> >>>>>>> would need a similar validation.
> > >> >>>>>>>
> > >> >>>>>>> Konstantine
> > >> >>>>>>>
> > >> >>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> > >> >>>>>>> <rd...@confluent.io.invalid> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Hey Konstantine,
> > >> >>>>>>>>
> > >> >>>>>>>> Thanks for the question. If these configs are not validated
> the
> > >> >>>>> user's
> > >> >>>>>>>> experience will be affected and upgrades from 3.0 will be
> > >> >> harder.
> > >> >>>>>>>>
> > >> >>>>>>>> Best,
> > >> >>>>>>>> Ryan Dielhenn
> > >> >>>>>>>>
> > >> >>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> > >> >>>>>>>> kkarantasis@apache.org> wrote:
> > >> >>>>>>>>
> > >> >>>>>>>>> Thanks for reporting this issue Ryan.
> > >> >>>>>>>>>
> > >> >>>>>>>>> I believe what you mention corresponds to the ticket you
> > >> >> created
> > >> >>>>> here:
> > >> >>>>>>>>>
> > >> >>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> > >> >>>>>>>>>
> > >> >>>>>>>>> What happens if the configurations are present but the
> broker
> > >> >>>>> doesn't
> > >> >>>>>>>> fail
> > >> >>>>>>>>> at startup when configured to run in KRaft mode?
> > >> >>>>>>>>> Asking to see if we have any workarounds in our
> availability.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>> Konstantine
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> > >> >>>>>>>>> <rd...@confluent.io.invalid> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> Hi,
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Disregard log.clean.policy being included in this blocker.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Best,
> > >> >>>>>>>>>> Ryan Dielhenn
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> > >> >>>>>>> rdielhenn@confluent.io>
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>> Hey Konstantine,
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I'd like to report another bug in KRaft.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
> > >> >>>>>>>>>>> create.topic.policy.class.name are all unsupported by
> > >> >> KRaft
> > >> >>>>> but
> > >> >>>>>>>> KRaft
> > >> >>>>>>>>>>> servers allow them to be configured. I believe this should
> > >> >>> be
> > >> >>>>>>>>> considered
> > >> >>>>>>>>>> a
> > >> >>>>>>>>>>> blocker and that KRaft servers should fail startup if any
> > >> >> of
> > >> >>>>> these
> > >> >>>>>>>> are
> > >> >>>>>>>>>>> configured. I do not have a PR yet but will soon.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On another note, I have a PR for the dynamic broker
> > >> >>>>> configuration
> > >> >>>>>>> fix
> > >> >>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Best,
> > >> >>>>>>>>>>> Ryan Dielhenn
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > >> >>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>> Hi all,
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Please find below the updated release plan for the Apache
> > >> >>>>> Kafka
> > >> >>>>>>>> 3.0.0
> > >> >>>>>>>>>>>> release.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>
> > >> >>>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> New suggested dates for the release are as follows:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
> > >> >>> plan)
> > >> >>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
> > >> >>>>> weeks)
> > >> >>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
> > >> >>> weeks)
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> At least two weeks of stabilization will follow Code
> > >> >>> Freeze.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> The release plan is up to date and currently includes all
> > >> >>> the
> > >> >>>>>>>> approved
> > >> >>>>>>>>>>>> KIPs
> > >> >>>>>>>>>>>> that are targeting 3.0.0.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Please let me know if you have any objections with the
> > >> >>> recent
> > >> >>>>>>>>> extension
> > >> >>>>>>>>>> of
> > >> >>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Regards,
> > >> >>>>>>>>>>>> Konstantine
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> --
> > >> >>>>> -Jose
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Ron Dagostino <rn...@gmail.com>.
Hi Konstantine. A potential 3.0 blocker was discovered today,
https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13219. The
BrokerState metric is not working for KRaft clusters -- it always indicates
that the broker is in the "not running" state. A PR is available at
https://github.com/apache/kafka/pull/11239. The risk is low, and there is
no workaround, but I also understand the bar for a blocker is pretty high
at this point.
Ron
On Thu, Aug 19, 2021 at 5:54 PM Israel Ekpo <is...@gmail.com> wrote:
> Konstantine,
>
> After a brief chat with Sophie, I just updated the name for KIP-633 to be
> more descriptive of what is actually happening in the implementation
>
> It is changed on the KIP page and the JIRA task title is also updated to be
> more descriptive
> https://cwiki.apache.org/confluence/x/Ho2NCg
> https://issues.apache.org/jira/browse/KAFKA-8613
>
> However, I could not update the page for the 3.0 release plan. I do not
> have the permissions for that update.
> https://cwiki.apache.org/confluence/x/woONCg
>
> When you have a moment, please could you update the release plan for 3.0 to
> reflect the update name for the KIP?
>
> Thank you very much
>
> Sincerely,
> Israel
>
>
> On Thu, Aug 19, 2021 at 5:50 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Hi Matthias, it is possible to still deprecate JoinWindows.of(size) even
> > though the new join semantics is disabled.
> >
> > I just pre-recorded a talk for Kafka Summit Americas where I am
> > recommending a switch to the new APIs instead of the deprecated one
> > starting from 3.0
> >
> > I would love to be involved in the discussion for the fix so that we try
> > to honor the expectations of the KIP as much as possible.
> >
> > So far, it looks like the PR will still honor the expectations of the KIP
> >
> > https://github.com/apache/kafka/pull/11235
> >
> > Thanks for sharing this to create awareness.
> >
> > Sincerely,
> > Israel
> >
> >
> >
> >
> >
> > On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org>
> wrote:
> >
> >> Hi,
> >>
> >> we discovered a potential blocker for 3.0.0 today:
> >> https://issues.apache.org/jira/browse/KAFKA-13216
> >>
> >> We are still evaluating a potential fix. If we cannot fix it quickly,
> >> the fall-back would be to partially roll-back KIP-633, to disable the
> >> new join semantics such that people cannot hit this bug.
> >>
> >> Thoughts?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
> >> > Hi Rajini.
> >> >
> >> > Approved, given its low risk and the lack of convenient workarounds.
> >> >
> >> > Konstantine
> >> >
> >> >
> >> > On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <
> >> rajinisivaram@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi Konstantine,
> >> >>
> >> >> We found an issue with replication with IBP 2.7:
> >> >> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small
> >> and
> >> >> low
> >> >> risk and has been merged to trunk. Can we include this in 3.0 branch
> >> since
> >> >> it can result in IllegalStateException during replication?
> >> >>
> >> >> Thank you,
> >> >>
> >> >> Rajini
> >> >>
> >> >>
> >> >> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
> >> >> kkarantasis@apache.org> wrote:
> >> >>
> >> >>> Jose, thanks for the heads up on the 3 new blocker candidates.
> >> >>>
> >> >>> I read the tickets and they have clear descriptions and
> implementation
> >> >>> details.
> >> >>> However, at this stage to be able to make a call and approve new
> >> blockers
> >> >>> I'd appreciate it if we could get some insight regarding the risk
> and
> >> the
> >> >>> necessity of a fix. A rough ETA would also be helpful.
> >> >>>
> >> >>> Having said that, based on the descriptions and the existence of a
> few
> >> >>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165,
> >> and
> >> >>> KAFKA-13168 and we might have to make a new assessment if these are
> >> the
> >> >>> only blockers in the next few days or if we notice a regression
> during
> >> >>> testing.
> >> >>>
> >> >>> Konstantine
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
> >> >>> kkarantasis@apache.org> wrote:
> >> >>>
> >> >>>>
> >> >>>> Thanks for reporting this new issue Ryan,
> >> >>>>
> >> >>>> It's important and this issue seems to have clearly regressed
> dynamic
> >> >>>> default configs in the 3.0 branch.
> >> >>>> So, it's approved.
> >> >>>>
> >> >>>> Konstantine
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> >> >>>> <js...@confluent.io.invalid> wrote:
> >> >>>>
> >> >>>>> Hey all,
> >> >>>>>
> >> >>>>> For the KIP-500 work for 3.0 we would like to propose the
> following
> >> >>>>> Jiras as blockers:
> >> >>>>>
> >> >>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> >> >>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> >> >>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
> >> >>>>>
> >> >>>>> The description for each Jira should have more details.
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> -Jose
> >> >>>>>
> >> >>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> >> >>>>> <rd...@confluent.io.invalid> wrote:
> >> >>>>>>
> >> >>>>>> Hi Konstantine,
> >> >>>>>>
> >> >>>>>> I would like to report another bug in KRaft.
> >> >>>>>>
> >> >>>>>> The ConfigHandler that processes dynamic broker config deltas in
> >> >> KRaft
> >> >>>>>> expects that the default resource name for dynamic broker configs
> >> is
> >> >>> the
> >> >>>>>> old default entity name used in ZK: "<default>". Since dynamic
> >> >> default
> >> >>>>>> broker configs are persisted as empty string in the quorum
> instead
> >> >> of
> >> >>>>>> "<default>", the brokers are not updating the their default
> >> >>>>> configuration
> >> >>>>>> when they see empty string as a resource name in the config delta
> >> >> and
> >> >>>>> are
> >> >>>>>> throwing a NumberFormatException when they try to parse the
> >> resource
> >> >>>>> name
> >> >>>>>> to process it as a per-broker configuration.
> >> >>>>>>
> >> >>>>>> I filed a JIRA:
> https://issues.apache.org/jira/browse/KAFKA-13160
> >> >>>>>>
> >> >>>>>> I also have a PR to fix this:
> >> >>>>> https://github.com/apache/kafka/pull/11168
> >> >>>>>>
> >> >>>>>> I think that this should be a blocker for 3.0 because dynamic
> >> >> default
> >> >>>>>> broker configs will not be usable in KRaft otherwise.
> >> >>>>>>
> >> >>>>>> Best,
> >> >>>>>> Ryan Dielhenn
> >> >>>>>>
> >> >>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> >> >>>>>> kkarantasis@apache.org> wrote:
> >> >>>>>>
> >> >>>>>>> Thanks Ryan,
> >> >>>>>>>
> >> >>>>>>> Approved. Seems also like a low risk fix.
> >> >>>>>>> With that opportunity, let's make sure there are no other
> configs
> >> >>> that
> >> >>>>>>> would need a similar validation.
> >> >>>>>>>
> >> >>>>>>> Konstantine
> >> >>>>>>>
> >> >>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> >> >>>>>>> <rd...@confluent.io.invalid> wrote:
> >> >>>>>>>
> >> >>>>>>>> Hey Konstantine,
> >> >>>>>>>>
> >> >>>>>>>> Thanks for the question. If these configs are not validated the
> >> >>>>> user's
> >> >>>>>>>> experience will be affected and upgrades from 3.0 will be
> >> >> harder.
> >> >>>>>>>>
> >> >>>>>>>> Best,
> >> >>>>>>>> Ryan Dielhenn
> >> >>>>>>>>
> >> >>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> >> >>>>>>>> kkarantasis@apache.org> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> Thanks for reporting this issue Ryan.
> >> >>>>>>>>>
> >> >>>>>>>>> I believe what you mention corresponds to the ticket you
> >> >> created
> >> >>>>> here:
> >> >>>>>>>>>
> >> >>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> >> >>>>>>>>>
> >> >>>>>>>>> What happens if the configurations are present but the broker
> >> >>>>> doesn't
> >> >>>>>>>> fail
> >> >>>>>>>>> at startup when configured to run in KRaft mode?
> >> >>>>>>>>> Asking to see if we have any workarounds in our availability.
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks,
> >> >>>>>>>>> Konstantine
> >> >>>>>>>>>
> >> >>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> >> >>>>>>>>> <rd...@confluent.io.invalid> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> Hi,
> >> >>>>>>>>>>
> >> >>>>>>>>>> Disregard log.clean.policy being included in this blocker.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Best,
> >> >>>>>>>>>> Ryan Dielhenn
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> >> >>>>>>> rdielhenn@confluent.io>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>>> Hey Konstantine,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> I'd like to report another bug in KRaft.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
> >> >>>>>>>>>>> create.topic.policy.class.name are all unsupported by
> >> >> KRaft
> >> >>>>> but
> >> >>>>>>>> KRaft
> >> >>>>>>>>>>> servers allow them to be configured. I believe this should
> >> >>> be
> >> >>>>>>>>> considered
> >> >>>>>>>>>> a
> >> >>>>>>>>>>> blocker and that KRaft servers should fail startup if any
> >> >> of
> >> >>>>> these
> >> >>>>>>>> are
> >> >>>>>>>>>>> configured. I do not have a PR yet but will soon.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On another note, I have a PR for the dynamic broker
> >> >>>>> configuration
> >> >>>>>>> fix
> >> >>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Best,
> >> >>>>>>>>>>> Ryan Dielhenn
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> >> >>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Hi all,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Please find below the updated release plan for the Apache
> >> >>>>> Kafka
> >> >>>>>>>> 3.0.0
> >> >>>>>>>>>>>> release.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> New suggested dates for the release are as follows:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
> >> >>> plan)
> >> >>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
> >> >>>>> weeks)
> >> >>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
> >> >>> weeks)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> At least two weeks of stabilization will follow Code
> >> >>> Freeze.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> The release plan is up to date and currently includes all
> >> >>> the
> >> >>>>>>>> approved
> >> >>>>>>>>>>>> KIPs
> >> >>>>>>>>>>>> that are targeting 3.0.0.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Please let me know if you have any objections with the
> >> >>> recent
> >> >>>>>>>>> extension
> >> >>>>>>>>>> of
> >> >>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Regards,
> >> >>>>>>>>>>>> Konstantine
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> -Jose
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by "Matthias J. Sax" <mj...@apache.org>.
Given the discussed upgrade path from the current join semantics to the
improved join semantics, I think it's best to partly revert KIP-633 for
3.0 release.
I prepared a PR for it: https://github.com/apache/kafka/pull/11233
The idea is to stay on `JoinWindows.of().grace()` and not add
`JoinWindows.ofTimeDifferenceAndGrace / NoGrace()`: this way users
cannot enable the spurious left/outer join fix and we are safe.
We can fix the join bug and complete KIP-633 in 3.1 release.
As we undo the deprecation for `JoinWindows.of()` and
`JoinWindows.grace()`, I assume your talk is still correct... I actually
also talked about the stream/stream join fix in Kafka Summit Europe this
yeer "advertising" the improved join semantics... So we are somewhat in
the same boat. Still think it's fine if it only ships in 3.1 instead of 3.0.
I understand that it is annoying to only ship a partially implemented
KIP, but I strongly believe it's the best way forward.
-Matthias
On 8/19/21 2:53 PM, Israel Ekpo wrote:
> Konstantine,
>
> After a brief chat with Sophie, I just updated the name for KIP-633 to be
> more descriptive of what is actually happening in the implementation
>
> It is changed on the KIP page and the JIRA task title is also updated to be
> more descriptive
> https://cwiki.apache.org/confluence/x/Ho2NCg
> https://issues.apache.org/jira/browse/KAFKA-8613
>
> However, I could not update the page for the 3.0 release plan. I do not
> have the permissions for that update.
> https://cwiki.apache.org/confluence/x/woONCg
>
> When you have a moment, please could you update the release plan for 3.0 to
> reflect the update name for the KIP?
>
> Thank you very much
>
> Sincerely,
> Israel
>
>
> On Thu, Aug 19, 2021 at 5:50 PM Israel Ekpo <is...@gmail.com> wrote:
>
>> Hi Matthias, it is possible to still deprecate JoinWindows.of(size) even
>> though the new join semantics is disabled.
>>
>> I just pre-recorded a talk for Kafka Summit Americas where I am
>> recommending a switch to the new APIs instead of the deprecated one
>> starting from 3.0
>>
>> I would love to be involved in the discussion for the fix so that we try
>> to honor the expectations of the KIP as much as possible.
>>
>> So far, it looks like the PR will still honor the expectations of the KIP
>>
>> https://github.com/apache/kafka/pull/11235
>>
>> Thanks for sharing this to create awareness.
>>
>> Sincerely,
>> Israel
>>
>>
>>
>>
>>
>> On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> we discovered a potential blocker for 3.0.0 today:
>>> https://issues.apache.org/jira/browse/KAFKA-13216
>>>
>>> We are still evaluating a potential fix. If we cannot fix it quickly,
>>> the fall-back would be to partially roll-back KIP-633, to disable the
>>> new join semantics such that people cannot hit this bug.
>>>
>>> Thoughts?
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
>>>> Hi Rajini.
>>>>
>>>> Approved, given its low risk and the lack of convenient workarounds.
>>>>
>>>> Konstantine
>>>>
>>>>
>>>> On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <
>>> rajinisivaram@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Konstantine,
>>>>>
>>>>> We found an issue with replication with IBP 2.7:
>>>>> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small
>>> and
>>>>> low
>>>>> risk and has been merged to trunk. Can we include this in 3.0 branch
>>> since
>>>>> it can result in IllegalStateException during replication?
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Rajini
>>>>>
>>>>>
>>>>> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
>>>>> kkarantasis@apache.org> wrote:
>>>>>
>>>>>> Jose, thanks for the heads up on the 3 new blocker candidates.
>>>>>>
>>>>>> I read the tickets and they have clear descriptions and implementation
>>>>>> details.
>>>>>> However, at this stage to be able to make a call and approve new
>>> blockers
>>>>>> I'd appreciate it if we could get some insight regarding the risk and
>>> the
>>>>>> necessity of a fix. A rough ETA would also be helpful.
>>>>>>
>>>>>> Having said that, based on the descriptions and the existence of a few
>>>>>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165,
>>> and
>>>>>> KAFKA-13168 and we might have to make a new assessment if these are
>>> the
>>>>>> only blockers in the next few days or if we notice a regression during
>>>>>> testing.
>>>>>>
>>>>>> Konstantine
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
>>>>>> kkarantasis@apache.org> wrote:
>>>>>>
>>>>>>>
>>>>>>> Thanks for reporting this new issue Ryan,
>>>>>>>
>>>>>>> It's important and this issue seems to have clearly regressed dynamic
>>>>>>> default configs in the 3.0 branch.
>>>>>>> So, it's approved.
>>>>>>>
>>>>>>> Konstantine
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
>>>>>>> <js...@confluent.io.invalid> wrote:
>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> For the KIP-500 work for 3.0 we would like to propose the following
>>>>>>>> Jiras as blockers:
>>>>>>>>
>>>>>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
>>>>>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
>>>>>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
>>>>>>>>
>>>>>>>> The description for each Jira should have more details.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Jose
>>>>>>>>
>>>>>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
>>>>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>>>>
>>>>>>>>> Hi Konstantine,
>>>>>>>>>
>>>>>>>>> I would like to report another bug in KRaft.
>>>>>>>>>
>>>>>>>>> The ConfigHandler that processes dynamic broker config deltas in
>>>>> KRaft
>>>>>>>>> expects that the default resource name for dynamic broker configs
>>> is
>>>>>> the
>>>>>>>>> old default entity name used in ZK: "<default>". Since dynamic
>>>>> default
>>>>>>>>> broker configs are persisted as empty string in the quorum instead
>>>>> of
>>>>>>>>> "<default>", the brokers are not updating the their default
>>>>>>>> configuration
>>>>>>>>> when they see empty string as a resource name in the config delta
>>>>> and
>>>>>>>> are
>>>>>>>>> throwing a NumberFormatException when they try to parse the
>>> resource
>>>>>>>> name
>>>>>>>>> to process it as a per-broker configuration.
>>>>>>>>>
>>>>>>>>> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
>>>>>>>>>
>>>>>>>>> I also have a PR to fix this:
>>>>>>>> https://github.com/apache/kafka/pull/11168
>>>>>>>>>
>>>>>>>>> I think that this should be a blocker for 3.0 because dynamic
>>>>> default
>>>>>>>>> broker configs will not be usable in KRaft otherwise.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Ryan Dielhenn
>>>>>>>>>
>>>>>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
>>>>>>>>> kkarantasis@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Ryan,
>>>>>>>>>>
>>>>>>>>>> Approved. Seems also like a low risk fix.
>>>>>>>>>> With that opportunity, let's make sure there are no other configs
>>>>>> that
>>>>>>>>>> would need a similar validation.
>>>>>>>>>>
>>>>>>>>>> Konstantine
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
>>>>>>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Konstantine,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the question. If these configs are not validated the
>>>>>>>> user's
>>>>>>>>>>> experience will be affected and upgrades from 3.0 will be
>>>>> harder.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Ryan Dielhenn
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
>>>>>>>>>>> kkarantasis@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for reporting this issue Ryan.
>>>>>>>>>>>>
>>>>>>>>>>>> I believe what you mention corresponds to the ticket you
>>>>> created
>>>>>>>> here:
>>>>>>>>>>>>
>>>>>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
>>>>>>>>>>>>
>>>>>>>>>>>> What happens if the configurations are present but the broker
>>>>>>>> doesn't
>>>>>>>>>>> fail
>>>>>>>>>>>> at startup when configured to run in KRaft mode?
>>>>>>>>>>>> Asking to see if we have any workarounds in our availability.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Konstantine
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
>>>>>>>>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Disregard log.clean.policy being included in this blocker.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Ryan Dielhenn
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
>>>>>>>>>> rdielhenn@confluent.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Konstantine,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd like to report another bug in KRaft.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
>>>>>>>>>>>>>> create.topic.policy.class.name are all unsupported by
>>>>> KRaft
>>>>>>>> but
>>>>>>>>>>> KRaft
>>>>>>>>>>>>>> servers allow them to be configured. I believe this should
>>>>>> be
>>>>>>>>>>>> considered
>>>>>>>>>>>>> a
>>>>>>>>>>>>>> blocker and that KRaft servers should fail startup if any
>>>>> of
>>>>>>>> these
>>>>>>>>>>> are
>>>>>>>>>>>>>> configured. I do not have a PR yet but will soon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On another note, I have a PR for the dynamic broker
>>>>>>>> configuration
>>>>>>>>>> fix
>>>>>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Ryan Dielhenn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>>>>>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please find below the updated release plan for the Apache
>>>>>>>> Kafka
>>>>>>>>>>> 3.0.0
>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> New suggested dates for the release are as follows:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
>>>>>> plan)
>>>>>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
>>>>>>>> weeks)
>>>>>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
>>>>>> weeks)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At least two weeks of stabilization will follow Code
>>>>>> Freeze.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The release plan is up to date and currently includes all
>>>>>> the
>>>>>>>>>>> approved
>>>>>>>>>>>>>>> KIPs
>>>>>>>>>>>>>>> that are targeting 3.0.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let me know if you have any objections with the
>>>>>> recent
>>>>>>>>>>>> extension
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Konstantine
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Jose
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Israel Ekpo <is...@gmail.com>.
Konstantine,
After a brief chat with Sophie, I just updated the name for KIP-633 to be
more descriptive of what is actually happening in the implementation
It is changed on the KIP page and the JIRA task title is also updated to be
more descriptive
https://cwiki.apache.org/confluence/x/Ho2NCg
https://issues.apache.org/jira/browse/KAFKA-8613
However, I could not update the page for the 3.0 release plan. I do not
have the permissions for that update.
https://cwiki.apache.org/confluence/x/woONCg
When you have a moment, please could you update the release plan for 3.0 to
reflect the update name for the KIP?
Thank you very much
Sincerely,
Israel
On Thu, Aug 19, 2021 at 5:50 PM Israel Ekpo <is...@gmail.com> wrote:
> Hi Matthias, it is possible to still deprecate JoinWindows.of(size) even
> though the new join semantics is disabled.
>
> I just pre-recorded a talk for Kafka Summit Americas where I am
> recommending a switch to the new APIs instead of the deprecated one
> starting from 3.0
>
> I would love to be involved in the discussion for the fix so that we try
> to honor the expectations of the KIP as much as possible.
>
> So far, it looks like the PR will still honor the expectations of the KIP
>
> https://github.com/apache/kafka/pull/11235
>
> Thanks for sharing this to create awareness.
>
> Sincerely,
> Israel
>
>
>
>
>
> On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org> wrote:
>
>> Hi,
>>
>> we discovered a potential blocker for 3.0.0 today:
>> https://issues.apache.org/jira/browse/KAFKA-13216
>>
>> We are still evaluating a potential fix. If we cannot fix it quickly,
>> the fall-back would be to partially roll-back KIP-633, to disable the
>> new join semantics such that people cannot hit this bug.
>>
>> Thoughts?
>>
>>
>> -Matthias
>>
>>
>> On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
>> > Hi Rajini.
>> >
>> > Approved, given its low risk and the lack of convenient workarounds.
>> >
>> > Konstantine
>> >
>> >
>> > On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <
>> rajinisivaram@gmail.com>
>> > wrote:
>> >
>> >> Hi Konstantine,
>> >>
>> >> We found an issue with replication with IBP 2.7:
>> >> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small
>> and
>> >> low
>> >> risk and has been merged to trunk. Can we include this in 3.0 branch
>> since
>> >> it can result in IllegalStateException during replication?
>> >>
>> >> Thank you,
>> >>
>> >> Rajini
>> >>
>> >>
>> >> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
>> >> kkarantasis@apache.org> wrote:
>> >>
>> >>> Jose, thanks for the heads up on the 3 new blocker candidates.
>> >>>
>> >>> I read the tickets and they have clear descriptions and implementation
>> >>> details.
>> >>> However, at this stage to be able to make a call and approve new
>> blockers
>> >>> I'd appreciate it if we could get some insight regarding the risk and
>> the
>> >>> necessity of a fix. A rough ETA would also be helpful.
>> >>>
>> >>> Having said that, based on the descriptions and the existence of a few
>> >>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165,
>> and
>> >>> KAFKA-13168 and we might have to make a new assessment if these are
>> the
>> >>> only blockers in the next few days or if we notice a regression during
>> >>> testing.
>> >>>
>> >>> Konstantine
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
>> >>> kkarantasis@apache.org> wrote:
>> >>>
>> >>>>
>> >>>> Thanks for reporting this new issue Ryan,
>> >>>>
>> >>>> It's important and this issue seems to have clearly regressed dynamic
>> >>>> default configs in the 3.0 branch.
>> >>>> So, it's approved.
>> >>>>
>> >>>> Konstantine
>> >>>>
>> >>>>
>> >>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
>> >>>> <js...@confluent.io.invalid> wrote:
>> >>>>
>> >>>>> Hey all,
>> >>>>>
>> >>>>> For the KIP-500 work for 3.0 we would like to propose the following
>> >>>>> Jiras as blockers:
>> >>>>>
>> >>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
>> >>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
>> >>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
>> >>>>>
>> >>>>> The description for each Jira should have more details.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> -Jose
>> >>>>>
>> >>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
>> >>>>> <rd...@confluent.io.invalid> wrote:
>> >>>>>>
>> >>>>>> Hi Konstantine,
>> >>>>>>
>> >>>>>> I would like to report another bug in KRaft.
>> >>>>>>
>> >>>>>> The ConfigHandler that processes dynamic broker config deltas in
>> >> KRaft
>> >>>>>> expects that the default resource name for dynamic broker configs
>> is
>> >>> the
>> >>>>>> old default entity name used in ZK: "<default>". Since dynamic
>> >> default
>> >>>>>> broker configs are persisted as empty string in the quorum instead
>> >> of
>> >>>>>> "<default>", the brokers are not updating the their default
>> >>>>> configuration
>> >>>>>> when they see empty string as a resource name in the config delta
>> >> and
>> >>>>> are
>> >>>>>> throwing a NumberFormatException when they try to parse the
>> resource
>> >>>>> name
>> >>>>>> to process it as a per-broker configuration.
>> >>>>>>
>> >>>>>> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
>> >>>>>>
>> >>>>>> I also have a PR to fix this:
>> >>>>> https://github.com/apache/kafka/pull/11168
>> >>>>>>
>> >>>>>> I think that this should be a blocker for 3.0 because dynamic
>> >> default
>> >>>>>> broker configs will not be usable in KRaft otherwise.
>> >>>>>>
>> >>>>>> Best,
>> >>>>>> Ryan Dielhenn
>> >>>>>>
>> >>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
>> >>>>>> kkarantasis@apache.org> wrote:
>> >>>>>>
>> >>>>>>> Thanks Ryan,
>> >>>>>>>
>> >>>>>>> Approved. Seems also like a low risk fix.
>> >>>>>>> With that opportunity, let's make sure there are no other configs
>> >>> that
>> >>>>>>> would need a similar validation.
>> >>>>>>>
>> >>>>>>> Konstantine
>> >>>>>>>
>> >>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
>> >>>>>>> <rd...@confluent.io.invalid> wrote:
>> >>>>>>>
>> >>>>>>>> Hey Konstantine,
>> >>>>>>>>
>> >>>>>>>> Thanks for the question. If these configs are not validated the
>> >>>>> user's
>> >>>>>>>> experience will be affected and upgrades from 3.0 will be
>> >> harder.
>> >>>>>>>>
>> >>>>>>>> Best,
>> >>>>>>>> Ryan Dielhenn
>> >>>>>>>>
>> >>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
>> >>>>>>>> kkarantasis@apache.org> wrote:
>> >>>>>>>>
>> >>>>>>>>> Thanks for reporting this issue Ryan.
>> >>>>>>>>>
>> >>>>>>>>> I believe what you mention corresponds to the ticket you
>> >> created
>> >>>>> here:
>> >>>>>>>>>
>> >>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
>> >>>>>>>>>
>> >>>>>>>>> What happens if the configurations are present but the broker
>> >>>>> doesn't
>> >>>>>>>> fail
>> >>>>>>>>> at startup when configured to run in KRaft mode?
>> >>>>>>>>> Asking to see if we have any workarounds in our availability.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> Konstantine
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
>> >>>>>>>>> <rd...@confluent.io.invalid> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hi,
>> >>>>>>>>>>
>> >>>>>>>>>> Disregard log.clean.policy being included in this blocker.
>> >>>>>>>>>>
>> >>>>>>>>>> Best,
>> >>>>>>>>>> Ryan Dielhenn
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
>> >>>>>>> rdielhenn@confluent.io>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hey Konstantine,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'd like to report another bug in KRaft.
>> >>>>>>>>>>>
>> >>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
>> >>>>>>>>>>> create.topic.policy.class.name are all unsupported by
>> >> KRaft
>> >>>>> but
>> >>>>>>>> KRaft
>> >>>>>>>>>>> servers allow them to be configured. I believe this should
>> >>> be
>> >>>>>>>>> considered
>> >>>>>>>>>> a
>> >>>>>>>>>>> blocker and that KRaft servers should fail startup if any
>> >> of
>> >>>>> these
>> >>>>>>>> are
>> >>>>>>>>>>> configured. I do not have a PR yet but will soon.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On another note, I have a PR for the dynamic broker
>> >>>>> configuration
>> >>>>>>> fix
>> >>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Ryan Dielhenn
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>> >>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi all,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Please find below the updated release plan for the Apache
>> >>>>> Kafka
>> >>>>>>>> 3.0.0
>> >>>>>>>>>>>> release.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> New suggested dates for the release are as follows:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
>> >>> plan)
>> >>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
>> >>>>> weeks)
>> >>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
>> >>> weeks)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> At least two weeks of stabilization will follow Code
>> >>> Freeze.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The release plan is up to date and currently includes all
>> >>> the
>> >>>>>>>> approved
>> >>>>>>>>>>>> KIPs
>> >>>>>>>>>>>> that are targeting 3.0.0.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Please let me know if you have any objections with the
>> >>> recent
>> >>>>>>>>> extension
>> >>>>>>>>>> of
>> >>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Regards,
>> >>>>>>>>>>>> Konstantine
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> -Jose
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Israel Ekpo <is...@gmail.com>.
Hi Matthias, it is possible to still deprecate JoinWindows.of(size) even
though the new join semantics is disabled.
I just pre-recorded a talk for Kafka Summit Americas where I am
recommending a switch to the new APIs instead of the deprecated one
starting from 3.0
I would love to be involved in the discussion for the fix so that we try to
honor the expectations of the KIP as much as possible.
So far, it looks like the PR will still honor the expectations of the KIP
https://github.com/apache/kafka/pull/11235
Thanks for sharing this to create awareness.
Sincerely,
Israel
On Wed, Aug 18, 2021 at 8:57 PM Matthias J. Sax <mj...@apache.org> wrote:
> Hi,
>
> we discovered a potential blocker for 3.0.0 today:
> https://issues.apache.org/jira/browse/KAFKA-13216
>
> We are still evaluating a potential fix. If we cannot fix it quickly,
> the fall-back would be to partially roll-back KIP-633, to disable the
> new join semantics such that people cannot hit this bug.
>
> Thoughts?
>
>
> -Matthias
>
>
> On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
> > Hi Rajini.
> >
> > Approved, given its low risk and the lack of convenient workarounds.
> >
> > Konstantine
> >
> >
> > On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> >> Hi Konstantine,
> >>
> >> We found an issue with replication with IBP 2.7:
> >> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small and
> >> low
> >> risk and has been merged to trunk. Can we include this in 3.0 branch
> since
> >> it can result in IllegalStateException during replication?
> >>
> >> Thank you,
> >>
> >> Rajini
> >>
> >>
> >> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
> >> kkarantasis@apache.org> wrote:
> >>
> >>> Jose, thanks for the heads up on the 3 new blocker candidates.
> >>>
> >>> I read the tickets and they have clear descriptions and implementation
> >>> details.
> >>> However, at this stage to be able to make a call and approve new
> blockers
> >>> I'd appreciate it if we could get some insight regarding the risk and
> the
> >>> necessity of a fix. A rough ETA would also be helpful.
> >>>
> >>> Having said that, based on the descriptions and the existence of a few
> >>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, and
> >>> KAFKA-13168 and we might have to make a new assessment if these are the
> >>> only blockers in the next few days or if we notice a regression during
> >>> testing.
> >>>
> >>> Konstantine
> >>>
> >>>
> >>>
> >>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
> >>> kkarantasis@apache.org> wrote:
> >>>
> >>>>
> >>>> Thanks for reporting this new issue Ryan,
> >>>>
> >>>> It's important and this issue seems to have clearly regressed dynamic
> >>>> default configs in the 3.0 branch.
> >>>> So, it's approved.
> >>>>
> >>>> Konstantine
> >>>>
> >>>>
> >>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> >>>> <js...@confluent.io.invalid> wrote:
> >>>>
> >>>>> Hey all,
> >>>>>
> >>>>> For the KIP-500 work for 3.0 we would like to propose the following
> >>>>> Jiras as blockers:
> >>>>>
> >>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> >>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> >>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
> >>>>>
> >>>>> The description for each Jira should have more details.
> >>>>>
> >>>>> Thanks,
> >>>>> -Jose
> >>>>>
> >>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> >>>>> <rd...@confluent.io.invalid> wrote:
> >>>>>>
> >>>>>> Hi Konstantine,
> >>>>>>
> >>>>>> I would like to report another bug in KRaft.
> >>>>>>
> >>>>>> The ConfigHandler that processes dynamic broker config deltas in
> >> KRaft
> >>>>>> expects that the default resource name for dynamic broker configs is
> >>> the
> >>>>>> old default entity name used in ZK: "<default>". Since dynamic
> >> default
> >>>>>> broker configs are persisted as empty string in the quorum instead
> >> of
> >>>>>> "<default>", the brokers are not updating the their default
> >>>>> configuration
> >>>>>> when they see empty string as a resource name in the config delta
> >> and
> >>>>> are
> >>>>>> throwing a NumberFormatException when they try to parse the resource
> >>>>> name
> >>>>>> to process it as a per-broker configuration.
> >>>>>>
> >>>>>> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
> >>>>>>
> >>>>>> I also have a PR to fix this:
> >>>>> https://github.com/apache/kafka/pull/11168
> >>>>>>
> >>>>>> I think that this should be a blocker for 3.0 because dynamic
> >> default
> >>>>>> broker configs will not be usable in KRaft otherwise.
> >>>>>>
> >>>>>> Best,
> >>>>>> Ryan Dielhenn
> >>>>>>
> >>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> >>>>>> kkarantasis@apache.org> wrote:
> >>>>>>
> >>>>>>> Thanks Ryan,
> >>>>>>>
> >>>>>>> Approved. Seems also like a low risk fix.
> >>>>>>> With that opportunity, let's make sure there are no other configs
> >>> that
> >>>>>>> would need a similar validation.
> >>>>>>>
> >>>>>>> Konstantine
> >>>>>>>
> >>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> >>>>>>> <rd...@confluent.io.invalid> wrote:
> >>>>>>>
> >>>>>>>> Hey Konstantine,
> >>>>>>>>
> >>>>>>>> Thanks for the question. If these configs are not validated the
> >>>>> user's
> >>>>>>>> experience will be affected and upgrades from 3.0 will be
> >> harder.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Ryan Dielhenn
> >>>>>>>>
> >>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> >>>>>>>> kkarantasis@apache.org> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for reporting this issue Ryan.
> >>>>>>>>>
> >>>>>>>>> I believe what you mention corresponds to the ticket you
> >> created
> >>>>> here:
> >>>>>>>>>
> >>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> >>>>>>>>>
> >>>>>>>>> What happens if the configurations are present but the broker
> >>>>> doesn't
> >>>>>>>> fail
> >>>>>>>>> at startup when configured to run in KRaft mode?
> >>>>>>>>> Asking to see if we have any workarounds in our availability.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Konstantine
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> >>>>>>>>> <rd...@confluent.io.invalid> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> Disregard log.clean.policy being included in this blocker.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Ryan Dielhenn
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> >>>>>>> rdielhenn@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey Konstantine,
> >>>>>>>>>>>
> >>>>>>>>>>> I'd like to report another bug in KRaft.
> >>>>>>>>>>>
> >>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
> >>>>>>>>>>> create.topic.policy.class.name are all unsupported by
> >> KRaft
> >>>>> but
> >>>>>>>> KRaft
> >>>>>>>>>>> servers allow them to be configured. I believe this should
> >>> be
> >>>>>>>>> considered
> >>>>>>>>>> a
> >>>>>>>>>>> blocker and that KRaft servers should fail startup if any
> >> of
> >>>>> these
> >>>>>>>> are
> >>>>>>>>>>> configured. I do not have a PR yet but will soon.
> >>>>>>>>>>>
> >>>>>>>>>>> On another note, I have a PR for the dynamic broker
> >>>>> configuration
> >>>>>>> fix
> >>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Ryan Dielhenn
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> >>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please find below the updated release plan for the Apache
> >>>>> Kafka
> >>>>>>>> 3.0.0
> >>>>>>>>>>>> release.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >>>>>>>>>>>>
> >>>>>>>>>>>> New suggested dates for the release are as follows:
> >>>>>>>>>>>>
> >>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
> >>> plan)
> >>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
> >>>>> weeks)
> >>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
> >>> weeks)
> >>>>>>>>>>>>
> >>>>>>>>>>>> At least two weeks of stabilization will follow Code
> >>> Freeze.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The release plan is up to date and currently includes all
> >>> the
> >>>>>>>> approved
> >>>>>>>>>>>> KIPs
> >>>>>>>>>>>> that are targeting 3.0.0.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please let me know if you have any objections with the
> >>> recent
> >>>>>>>>> extension
> >>>>>>>>>> of
> >>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Konstantine
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -Jose
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hi,
we discovered a potential blocker for 3.0.0 today:
https://issues.apache.org/jira/browse/KAFKA-13216
We are still evaluating a potential fix. If we cannot fix it quickly,
the fall-back would be to partially roll-back KIP-633, to disable the
new join semantics such that people cannot hit this bug.
Thoughts?
-Matthias
On 8/17/21 1:29 PM, Konstantine Karantasis wrote:
> Hi Rajini.
>
> Approved, given its low risk and the lack of convenient workarounds.
>
> Konstantine
>
>
> On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Konstantine,
>>
>> We found an issue with replication with IBP 2.7:
>> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small and
>> low
>> risk and has been merged to trunk. Can we include this in 3.0 branch since
>> it can result in IllegalStateException during replication?
>>
>> Thank you,
>>
>> Rajini
>>
>>
>> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
>> kkarantasis@apache.org> wrote:
>>
>>> Jose, thanks for the heads up on the 3 new blocker candidates.
>>>
>>> I read the tickets and they have clear descriptions and implementation
>>> details.
>>> However, at this stage to be able to make a call and approve new blockers
>>> I'd appreciate it if we could get some insight regarding the risk and the
>>> necessity of a fix. A rough ETA would also be helpful.
>>>
>>> Having said that, based on the descriptions and the existence of a few
>>> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, and
>>> KAFKA-13168 and we might have to make a new assessment if these are the
>>> only blockers in the next few days or if we notice a regression during
>>> testing.
>>>
>>> Konstantine
>>>
>>>
>>>
>>> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
>>> kkarantasis@apache.org> wrote:
>>>
>>>>
>>>> Thanks for reporting this new issue Ryan,
>>>>
>>>> It's important and this issue seems to have clearly regressed dynamic
>>>> default configs in the 3.0 branch.
>>>> So, it's approved.
>>>>
>>>> Konstantine
>>>>
>>>>
>>>> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
>>>> <js...@confluent.io.invalid> wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> For the KIP-500 work for 3.0 we would like to propose the following
>>>>> Jiras as blockers:
>>>>>
>>>>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
>>>>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
>>>>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
>>>>>
>>>>> The description for each Jira should have more details.
>>>>>
>>>>> Thanks,
>>>>> -Jose
>>>>>
>>>>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>
>>>>>> Hi Konstantine,
>>>>>>
>>>>>> I would like to report another bug in KRaft.
>>>>>>
>>>>>> The ConfigHandler that processes dynamic broker config deltas in
>> KRaft
>>>>>> expects that the default resource name for dynamic broker configs is
>>> the
>>>>>> old default entity name used in ZK: "<default>". Since dynamic
>> default
>>>>>> broker configs are persisted as empty string in the quorum instead
>> of
>>>>>> "<default>", the brokers are not updating the their default
>>>>> configuration
>>>>>> when they see empty string as a resource name in the config delta
>> and
>>>>> are
>>>>>> throwing a NumberFormatException when they try to parse the resource
>>>>> name
>>>>>> to process it as a per-broker configuration.
>>>>>>
>>>>>> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
>>>>>>
>>>>>> I also have a PR to fix this:
>>>>> https://github.com/apache/kafka/pull/11168
>>>>>>
>>>>>> I think that this should be a blocker for 3.0 because dynamic
>> default
>>>>>> broker configs will not be usable in KRaft otherwise.
>>>>>>
>>>>>> Best,
>>>>>> Ryan Dielhenn
>>>>>>
>>>>>> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
>>>>>> kkarantasis@apache.org> wrote:
>>>>>>
>>>>>>> Thanks Ryan,
>>>>>>>
>>>>>>> Approved. Seems also like a low risk fix.
>>>>>>> With that opportunity, let's make sure there are no other configs
>>> that
>>>>>>> would need a similar validation.
>>>>>>>
>>>>>>> Konstantine
>>>>>>>
>>>>>>> On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
>>>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>>
>>>>>>>> Hey Konstantine,
>>>>>>>>
>>>>>>>> Thanks for the question. If these configs are not validated the
>>>>> user's
>>>>>>>> experience will be affected and upgrades from 3.0 will be
>> harder.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Ryan Dielhenn
>>>>>>>>
>>>>>>>> On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
>>>>>>>> kkarantasis@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Thanks for reporting this issue Ryan.
>>>>>>>>>
>>>>>>>>> I believe what you mention corresponds to the ticket you
>> created
>>>>> here:
>>>>>>>>>
>>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
>>>>>>>>>
>>>>>>>>> What happens if the configurations are present but the broker
>>>>> doesn't
>>>>>>>> fail
>>>>>>>>> at startup when configured to run in KRaft mode?
>>>>>>>>> Asking to see if we have any workarounds in our availability.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Konstantine
>>>>>>>>>
>>>>>>>>> On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
>>>>>>>>> <rd...@confluent.io.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Disregard log.clean.policy being included in this blocker.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Ryan Dielhenn
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
>>>>>>> rdielhenn@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Konstantine,
>>>>>>>>>>>
>>>>>>>>>>> I'd like to report another bug in KRaft.
>>>>>>>>>>>
>>>>>>>>>>> log.cleanup.policy, alter.config.policy.class.name, and
>>>>>>>>>>> create.topic.policy.class.name are all unsupported by
>> KRaft
>>>>> but
>>>>>>>> KRaft
>>>>>>>>>>> servers allow them to be configured. I believe this should
>>> be
>>>>>>>>> considered
>>>>>>>>>> a
>>>>>>>>>>> blocker and that KRaft servers should fail startup if any
>> of
>>>>> these
>>>>>>>> are
>>>>>>>>>>> configured. I do not have a PR yet but will soon.
>>>>>>>>>>>
>>>>>>>>>>> On another note, I have a PR for the dynamic broker
>>>>> configuration
>>>>>>> fix
>>>>>>>>>>> here: https://github.com/apache/kafka/pull/11141
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Ryan Dielhenn
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>>>>>>>>>>> <ko...@confluent.io.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> Please find below the updated release plan for the Apache
>>>>> Kafka
>>>>>>>> 3.0.0
>>>>>>>>>>>> release.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>>>>>>>>>>>>
>>>>>>>>>>>> New suggested dates for the release are as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> KIP Freeze is 09 June 2021 (same date as in the initial
>>> plan)
>>>>>>>>>>>> Feature Freeze is 30 June 2021 (new date, extended by two
>>>>> weeks)
>>>>>>>>>>>> Code Freeze is 14 July 2021 (new date, extended by two
>>> weeks)
>>>>>>>>>>>>
>>>>>>>>>>>> At least two weeks of stabilization will follow Code
>>> Freeze.
>>>>>>>>>>>>
>>>>>>>>>>>> The release plan is up to date and currently includes all
>>> the
>>>>>>>> approved
>>>>>>>>>>>> KIPs
>>>>>>>>>>>> that are targeting 3.0.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you have any objections with the
>>> recent
>>>>>>>>> extension
>>>>>>>>>> of
>>>>>>>>>>>> Feature Freeze and Code Freeze or any other concerns.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Konstantine
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Jose
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Konstantine Karantasis <kk...@apache.org>.
Hi Rajini.
Approved, given its low risk and the lack of convenient workarounds.
Konstantine
On Tue, Aug 17, 2021 at 11:00 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Konstantine,
>
> We found an issue with replication with IBP 2.7:
> https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small and
> low
> risk and has been merged to trunk. Can we include this in 3.0 branch since
> it can result in IllegalStateException during replication?
>
> Thank you,
>
> Rajini
>
>
> On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Jose, thanks for the heads up on the 3 new blocker candidates.
> >
> > I read the tickets and they have clear descriptions and implementation
> > details.
> > However, at this stage to be able to make a call and approve new blockers
> > I'd appreciate it if we could get some insight regarding the risk and the
> > necessity of a fix. A rough ETA would also be helpful.
> >
> > Having said that, based on the descriptions and the existence of a few
> > other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, and
> > KAFKA-13168 and we might have to make a new assessment if these are the
> > only blockers in the next few days or if we notice a regression during
> > testing.
> >
> > Konstantine
> >
> >
> >
> > On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > >
> > > Thanks for reporting this new issue Ryan,
> > >
> > > It's important and this issue seems to have clearly regressed dynamic
> > > default configs in the 3.0 branch.
> > > So, it's approved.
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> > > <js...@confluent.io.invalid> wrote:
> > >
> > >> Hey all,
> > >>
> > >> For the KIP-500 work for 3.0 we would like to propose the following
> > >> Jiras as blockers:
> > >>
> > >> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> > >> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> > >> 3. https://issues.apache.org/jira/browse/KAFKA-13161
> > >>
> > >> The description for each Jira should have more details.
> > >>
> > >> Thanks,
> > >> -Jose
> > >>
> > >> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> > >> <rd...@confluent.io.invalid> wrote:
> > >> >
> > >> > Hi Konstantine,
> > >> >
> > >> > I would like to report another bug in KRaft.
> > >> >
> > >> > The ConfigHandler that processes dynamic broker config deltas in
> KRaft
> > >> > expects that the default resource name for dynamic broker configs is
> > the
> > >> > old default entity name used in ZK: "<default>". Since dynamic
> default
> > >> > broker configs are persisted as empty string in the quorum instead
> of
> > >> > "<default>", the brokers are not updating the their default
> > >> configuration
> > >> > when they see empty string as a resource name in the config delta
> and
> > >> are
> > >> > throwing a NumberFormatException when they try to parse the resource
> > >> name
> > >> > to process it as a per-broker configuration.
> > >> >
> > >> > I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
> > >> >
> > >> > I also have a PR to fix this:
> > >> https://github.com/apache/kafka/pull/11168
> > >> >
> > >> > I think that this should be a blocker for 3.0 because dynamic
> default
> > >> > broker configs will not be usable in KRaft otherwise.
> > >> >
> > >> > Best,
> > >> > Ryan Dielhenn
> > >> >
> > >> > On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> > >> > kkarantasis@apache.org> wrote:
> > >> >
> > >> > > Thanks Ryan,
> > >> > >
> > >> > > Approved. Seems also like a low risk fix.
> > >> > > With that opportunity, let's make sure there are no other configs
> > that
> > >> > > would need a similar validation.
> > >> > >
> > >> > > Konstantine
> > >> > >
> > >> > > On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> > >> > > <rd...@confluent.io.invalid> wrote:
> > >> > >
> > >> > > > Hey Konstantine,
> > >> > > >
> > >> > > > Thanks for the question. If these configs are not validated the
> > >> user's
> > >> > > > experience will be affected and upgrades from 3.0 will be
> harder.
> > >> > > >
> > >> > > > Best,
> > >> > > > Ryan Dielhenn
> > >> > > >
> > >> > > > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> > >> > > > kkarantasis@apache.org> wrote:
> > >> > > >
> > >> > > > > Thanks for reporting this issue Ryan.
> > >> > > > >
> > >> > > > > I believe what you mention corresponds to the ticket you
> created
> > >> here:
> > >> > > > >
> > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> > >> > > > >
> > >> > > > > What happens if the configurations are present but the broker
> > >> doesn't
> > >> > > > fail
> > >> > > > > at startup when configured to run in KRaft mode?
> > >> > > > > Asking to see if we have any workarounds in our availability.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Konstantine
> > >> > > > >
> > >> > > > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> > >> > > > > <rd...@confluent.io.invalid> wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > Disregard log.clean.policy being included in this blocker.
> > >> > > > > >
> > >> > > > > > Best,
> > >> > > > > > Ryan Dielhenn
> > >> > > > > >
> > >> > > > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> > >> > > rdielhenn@confluent.io>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hey Konstantine,
> > >> > > > > > >
> > >> > > > > > > I'd like to report another bug in KRaft.
> > >> > > > > > >
> > >> > > > > > > log.cleanup.policy, alter.config.policy.class.name, and
> > >> > > > > > > create.topic.policy.class.name are all unsupported by
> KRaft
> > >> but
> > >> > > > KRaft
> > >> > > > > > > servers allow them to be configured. I believe this should
> > be
> > >> > > > > considered
> > >> > > > > > a
> > >> > > > > > > blocker and that KRaft servers should fail startup if any
> of
> > >> these
> > >> > > > are
> > >> > > > > > > configured. I do not have a PR yet but will soon.
> > >> > > > > > >
> > >> > > > > > > On another note, I have a PR for the dynamic broker
> > >> configuration
> > >> > > fix
> > >> > > > > > > here: https://github.com/apache/kafka/pull/11141
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Ryan Dielhenn
> > >> > > > > > >
> > >> > > > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > >> > > > > > > <ko...@confluent.io.invalid> wrote:
> > >> > > > > > >
> > >> > > > > > >> Hi all,
> > >> > > > > > >>
> > >> > > > > > >> Please find below the updated release plan for the Apache
> > >> Kafka
> > >> > > > 3.0.0
> > >> > > > > > >> release.
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > >> > > > > > >>
> > >> > > > > > >> New suggested dates for the release are as follows:
> > >> > > > > > >>
> > >> > > > > > >> KIP Freeze is 09 June 2021 (same date as in the initial
> > plan)
> > >> > > > > > >> Feature Freeze is 30 June 2021 (new date, extended by two
> > >> weeks)
> > >> > > > > > >> Code Freeze is 14 July 2021 (new date, extended by two
> > weeks)
> > >> > > > > > >>
> > >> > > > > > >> At least two weeks of stabilization will follow Code
> > Freeze.
> > >> > > > > > >>
> > >> > > > > > >> The release plan is up to date and currently includes all
> > the
> > >> > > > approved
> > >> > > > > > >> KIPs
> > >> > > > > > >> that are targeting 3.0.0.
> > >> > > > > > >>
> > >> > > > > > >> Please let me know if you have any objections with the
> > recent
> > >> > > > > extension
> > >> > > > > > of
> > >> > > > > > >> Feature Freeze and Code Freeze or any other concerns.
> > >> > > > > > >>
> > >> > > > > > >> Regards,
> > >> > > > > > >> Konstantine
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >>
> > >>
> > >>
> > >> --
> > >> -Jose
> > >>
> > >
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Konstantine,
We found an issue with replication with IBP 2.7:
https://issues.apache.org/jira/browse/KAFKA-13207. The fix is small and low
risk and has been merged to trunk. Can we include this in 3.0 branch since
it can result in IllegalStateException during replication?
Thank you,
Rajini
On Thu, Aug 5, 2021 at 6:21 PM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
> Jose, thanks for the heads up on the 3 new blocker candidates.
>
> I read the tickets and they have clear descriptions and implementation
> details.
> However, at this stage to be able to make a call and approve new blockers
> I'd appreciate it if we could get some insight regarding the risk and the
> necessity of a fix. A rough ETA would also be helpful.
>
> Having said that, based on the descriptions and the existence of a few
> other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, and
> KAFKA-13168 and we might have to make a new assessment if these are the
> only blockers in the next few days or if we notice a regression during
> testing.
>
> Konstantine
>
>
>
> On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> >
> > Thanks for reporting this new issue Ryan,
> >
> > It's important and this issue seems to have clearly regressed dynamic
> > default configs in the 3.0 branch.
> > So, it's approved.
> >
> > Konstantine
> >
> >
> > On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> > <js...@confluent.io.invalid> wrote:
> >
> >> Hey all,
> >>
> >> For the KIP-500 work for 3.0 we would like to propose the following
> >> Jiras as blockers:
> >>
> >> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> >> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> >> 3. https://issues.apache.org/jira/browse/KAFKA-13161
> >>
> >> The description for each Jira should have more details.
> >>
> >> Thanks,
> >> -Jose
> >>
> >> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> >> <rd...@confluent.io.invalid> wrote:
> >> >
> >> > Hi Konstantine,
> >> >
> >> > I would like to report another bug in KRaft.
> >> >
> >> > The ConfigHandler that processes dynamic broker config deltas in KRaft
> >> > expects that the default resource name for dynamic broker configs is
> the
> >> > old default entity name used in ZK: "<default>". Since dynamic default
> >> > broker configs are persisted as empty string in the quorum instead of
> >> > "<default>", the brokers are not updating the their default
> >> configuration
> >> > when they see empty string as a resource name in the config delta and
> >> are
> >> > throwing a NumberFormatException when they try to parse the resource
> >> name
> >> > to process it as a per-broker configuration.
> >> >
> >> > I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
> >> >
> >> > I also have a PR to fix this:
> >> https://github.com/apache/kafka/pull/11168
> >> >
> >> > I think that this should be a blocker for 3.0 because dynamic default
> >> > broker configs will not be usable in KRaft otherwise.
> >> >
> >> > Best,
> >> > Ryan Dielhenn
> >> >
> >> > On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> >> > kkarantasis@apache.org> wrote:
> >> >
> >> > > Thanks Ryan,
> >> > >
> >> > > Approved. Seems also like a low risk fix.
> >> > > With that opportunity, let's make sure there are no other configs
> that
> >> > > would need a similar validation.
> >> > >
> >> > > Konstantine
> >> > >
> >> > > On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> >> > > <rd...@confluent.io.invalid> wrote:
> >> > >
> >> > > > Hey Konstantine,
> >> > > >
> >> > > > Thanks for the question. If these configs are not validated the
> >> user's
> >> > > > experience will be affected and upgrades from 3.0 will be harder.
> >> > > >
> >> > > > Best,
> >> > > > Ryan Dielhenn
> >> > > >
> >> > > > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> >> > > > kkarantasis@apache.org> wrote:
> >> > > >
> >> > > > > Thanks for reporting this issue Ryan.
> >> > > > >
> >> > > > > I believe what you mention corresponds to the ticket you created
> >> here:
> >> > > > >
> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> >> > > > >
> >> > > > > What happens if the configurations are present but the broker
> >> doesn't
> >> > > > fail
> >> > > > > at startup when configured to run in KRaft mode?
> >> > > > > Asking to see if we have any workarounds in our availability.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Konstantine
> >> > > > >
> >> > > > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> >> > > > > <rd...@confluent.io.invalid> wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > Disregard log.clean.policy being included in this blocker.
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Ryan Dielhenn
> >> > > > > >
> >> > > > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> >> > > rdielhenn@confluent.io>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hey Konstantine,
> >> > > > > > >
> >> > > > > > > I'd like to report another bug in KRaft.
> >> > > > > > >
> >> > > > > > > log.cleanup.policy, alter.config.policy.class.name, and
> >> > > > > > > create.topic.policy.class.name are all unsupported by KRaft
> >> but
> >> > > > KRaft
> >> > > > > > > servers allow them to be configured. I believe this should
> be
> >> > > > > considered
> >> > > > > > a
> >> > > > > > > blocker and that KRaft servers should fail startup if any of
> >> these
> >> > > > are
> >> > > > > > > configured. I do not have a PR yet but will soon.
> >> > > > > > >
> >> > > > > > > On another note, I have a PR for the dynamic broker
> >> configuration
> >> > > fix
> >> > > > > > > here: https://github.com/apache/kafka/pull/11141
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Ryan Dielhenn
> >> > > > > > >
> >> > > > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> >> > > > > > > <ko...@confluent.io.invalid> wrote:
> >> > > > > > >
> >> > > > > > >> Hi all,
> >> > > > > > >>
> >> > > > > > >> Please find below the updated release plan for the Apache
> >> Kafka
> >> > > > 3.0.0
> >> > > > > > >> release.
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >> > > > > > >>
> >> > > > > > >> New suggested dates for the release are as follows:
> >> > > > > > >>
> >> > > > > > >> KIP Freeze is 09 June 2021 (same date as in the initial
> plan)
> >> > > > > > >> Feature Freeze is 30 June 2021 (new date, extended by two
> >> weeks)
> >> > > > > > >> Code Freeze is 14 July 2021 (new date, extended by two
> weeks)
> >> > > > > > >>
> >> > > > > > >> At least two weeks of stabilization will follow Code
> Freeze.
> >> > > > > > >>
> >> > > > > > >> The release plan is up to date and currently includes all
> the
> >> > > > approved
> >> > > > > > >> KIPs
> >> > > > > > >> that are targeting 3.0.0.
> >> > > > > > >>
> >> > > > > > >> Please let me know if you have any objections with the
> recent
> >> > > > > extension
> >> > > > > > of
> >> > > > > > >> Feature Freeze and Code Freeze or any other concerns.
> >> > > > > > >>
> >> > > > > > >> Regards,
> >> > > > > > >> Konstantine
> >> > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >>
> >>
> >>
> >> --
> >> -Jose
> >>
> >
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Konstantine Karantasis <kk...@apache.org>.
Jose, thanks for the heads up on the 3 new blocker candidates.
I read the tickets and they have clear descriptions and implementation
details.
However, at this stage to be able to make a call and approve new blockers
I'd appreciate it if we could get some insight regarding the risk and the
necessity of a fix. A rough ETA would also be helpful.
Having said that, based on the descriptions and the existence of a few
other blockers, I'm tentatively approving KAFKA-13161, KAFKA-13165, and
KAFKA-13168 and we might have to make a new assessment if these are the
only blockers in the next few days or if we notice a regression during
testing.
Konstantine
On Thu, Aug 5, 2021 at 10:04 AM Konstantine Karantasis <
kkarantasis@apache.org> wrote:
>
> Thanks for reporting this new issue Ryan,
>
> It's important and this issue seems to have clearly regressed dynamic
> default configs in the 3.0 branch.
> So, it's approved.
>
> Konstantine
>
>
> On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
> <js...@confluent.io.invalid> wrote:
>
>> Hey all,
>>
>> For the KIP-500 work for 3.0 we would like to propose the following
>> Jiras as blockers:
>>
>> 1. https://issues.apache.org/jira/browse/KAFKA-13168
>> 2. https://issues.apache.org/jira/browse/KAFKA-13165
>> 3. https://issues.apache.org/jira/browse/KAFKA-13161
>>
>> The description for each Jira should have more details.
>>
>> Thanks,
>> -Jose
>>
>> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
>> <rd...@confluent.io.invalid> wrote:
>> >
>> > Hi Konstantine,
>> >
>> > I would like to report another bug in KRaft.
>> >
>> > The ConfigHandler that processes dynamic broker config deltas in KRaft
>> > expects that the default resource name for dynamic broker configs is the
>> > old default entity name used in ZK: "<default>". Since dynamic default
>> > broker configs are persisted as empty string in the quorum instead of
>> > "<default>", the brokers are not updating the their default
>> configuration
>> > when they see empty string as a resource name in the config delta and
>> are
>> > throwing a NumberFormatException when they try to parse the resource
>> name
>> > to process it as a per-broker configuration.
>> >
>> > I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
>> >
>> > I also have a PR to fix this:
>> https://github.com/apache/kafka/pull/11168
>> >
>> > I think that this should be a blocker for 3.0 because dynamic default
>> > broker configs will not be usable in KRaft otherwise.
>> >
>> > Best,
>> > Ryan Dielhenn
>> >
>> > On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
>> > kkarantasis@apache.org> wrote:
>> >
>> > > Thanks Ryan,
>> > >
>> > > Approved. Seems also like a low risk fix.
>> > > With that opportunity, let's make sure there are no other configs that
>> > > would need a similar validation.
>> > >
>> > > Konstantine
>> > >
>> > > On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
>> > > <rd...@confluent.io.invalid> wrote:
>> > >
>> > > > Hey Konstantine,
>> > > >
>> > > > Thanks for the question. If these configs are not validated the
>> user's
>> > > > experience will be affected and upgrades from 3.0 will be harder.
>> > > >
>> > > > Best,
>> > > > Ryan Dielhenn
>> > > >
>> > > > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
>> > > > kkarantasis@apache.org> wrote:
>> > > >
>> > > > > Thanks for reporting this issue Ryan.
>> > > > >
>> > > > > I believe what you mention corresponds to the ticket you created
>> here:
>> > > > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
>> > > > >
>> > > > > What happens if the configurations are present but the broker
>> doesn't
>> > > > fail
>> > > > > at startup when configured to run in KRaft mode?
>> > > > > Asking to see if we have any workarounds in our availability.
>> > > > >
>> > > > > Thanks,
>> > > > > Konstantine
>> > > > >
>> > > > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
>> > > > > <rd...@confluent.io.invalid> wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Disregard log.clean.policy being included in this blocker.
>> > > > > >
>> > > > > > Best,
>> > > > > > Ryan Dielhenn
>> > > > > >
>> > > > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
>> > > rdielhenn@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hey Konstantine,
>> > > > > > >
>> > > > > > > I'd like to report another bug in KRaft.
>> > > > > > >
>> > > > > > > log.cleanup.policy, alter.config.policy.class.name, and
>> > > > > > > create.topic.policy.class.name are all unsupported by KRaft
>> but
>> > > > KRaft
>> > > > > > > servers allow them to be configured. I believe this should be
>> > > > > considered
>> > > > > > a
>> > > > > > > blocker and that KRaft servers should fail startup if any of
>> these
>> > > > are
>> > > > > > > configured. I do not have a PR yet but will soon.
>> > > > > > >
>> > > > > > > On another note, I have a PR for the dynamic broker
>> configuration
>> > > fix
>> > > > > > > here: https://github.com/apache/kafka/pull/11141
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Ryan Dielhenn
>> > > > > > >
>> > > > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>> > > > > > > <ko...@confluent.io.invalid> wrote:
>> > > > > > >
>> > > > > > >> Hi all,
>> > > > > > >>
>> > > > > > >> Please find below the updated release plan for the Apache
>> Kafka
>> > > > 3.0.0
>> > > > > > >> release.
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>> > > > > > >>
>> > > > > > >> New suggested dates for the release are as follows:
>> > > > > > >>
>> > > > > > >> KIP Freeze is 09 June 2021 (same date as in the initial plan)
>> > > > > > >> Feature Freeze is 30 June 2021 (new date, extended by two
>> weeks)
>> > > > > > >> Code Freeze is 14 July 2021 (new date, extended by two weeks)
>> > > > > > >>
>> > > > > > >> At least two weeks of stabilization will follow Code Freeze.
>> > > > > > >>
>> > > > > > >> The release plan is up to date and currently includes all the
>> > > > approved
>> > > > > > >> KIPs
>> > > > > > >> that are targeting 3.0.0.
>> > > > > > >>
>> > > > > > >> Please let me know if you have any objections with the recent
>> > > > > extension
>> > > > > > of
>> > > > > > >> Feature Freeze and Code Freeze or any other concerns.
>> > > > > > >>
>> > > > > > >> Regards,
>> > > > > > >> Konstantine
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>>
>>
>>
>> --
>> -Jose
>>
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by Konstantine Karantasis <kk...@apache.org>.
Thanks for reporting this new issue Ryan,
It's important and this issue seems to have clearly regressed dynamic
default configs in the 3.0 branch.
So, it's approved.
Konstantine
On Wed, Aug 4, 2021 at 4:34 PM José Armando García Sancio
<js...@confluent.io.invalid> wrote:
> Hey all,
>
> For the KIP-500 work for 3.0 we would like to propose the following
> Jiras as blockers:
>
> 1. https://issues.apache.org/jira/browse/KAFKA-13168
> 2. https://issues.apache.org/jira/browse/KAFKA-13165
> 3. https://issues.apache.org/jira/browse/KAFKA-13161
>
> The description for each Jira should have more details.
>
> Thanks,
> -Jose
>
> On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
> <rd...@confluent.io.invalid> wrote:
> >
> > Hi Konstantine,
> >
> > I would like to report another bug in KRaft.
> >
> > The ConfigHandler that processes dynamic broker config deltas in KRaft
> > expects that the default resource name for dynamic broker configs is the
> > old default entity name used in ZK: "<default>". Since dynamic default
> > broker configs are persisted as empty string in the quorum instead of
> > "<default>", the brokers are not updating the their default configuration
> > when they see empty string as a resource name in the config delta and are
> > throwing a NumberFormatException when they try to parse the resource name
> > to process it as a per-broker configuration.
> >
> > I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
> >
> > I also have a PR to fix this: https://github.com/apache/kafka/pull/11168
> >
> > I think that this should be a blocker for 3.0 because dynamic default
> > broker configs will not be usable in KRaft otherwise.
> >
> > Best,
> > Ryan Dielhenn
> >
> > On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> > kkarantasis@apache.org> wrote:
> >
> > > Thanks Ryan,
> > >
> > > Approved. Seems also like a low risk fix.
> > > With that opportunity, let's make sure there are no other configs that
> > > would need a similar validation.
> > >
> > > Konstantine
> > >
> > > On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> > > <rd...@confluent.io.invalid> wrote:
> > >
> > > > Hey Konstantine,
> > > >
> > > > Thanks for the question. If these configs are not validated the
> user's
> > > > experience will be affected and upgrades from 3.0 will be harder.
> > > >
> > > > Best,
> > > > Ryan Dielhenn
> > > >
> > > > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> > > > kkarantasis@apache.org> wrote:
> > > >
> > > > > Thanks for reporting this issue Ryan.
> > > > >
> > > > > I believe what you mention corresponds to the ticket you created
> here:
> > > > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> > > > >
> > > > > What happens if the configurations are present but the broker
> doesn't
> > > > fail
> > > > > at startup when configured to run in KRaft mode?
> > > > > Asking to see if we have any workarounds in our availability.
> > > > >
> > > > > Thanks,
> > > > > Konstantine
> > > > >
> > > > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> > > > > <rd...@confluent.io.invalid> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Disregard log.clean.policy being included in this blocker.
> > > > > >
> > > > > > Best,
> > > > > > Ryan Dielhenn
> > > > > >
> > > > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> > > rdielhenn@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Konstantine,
> > > > > > >
> > > > > > > I'd like to report another bug in KRaft.
> > > > > > >
> > > > > > > log.cleanup.policy, alter.config.policy.class.name, and
> > > > > > > create.topic.policy.class.name are all unsupported by KRaft
> but
> > > > KRaft
> > > > > > > servers allow them to be configured. I believe this should be
> > > > > considered
> > > > > > a
> > > > > > > blocker and that KRaft servers should fail startup if any of
> these
> > > > are
> > > > > > > configured. I do not have a PR yet but will soon.
> > > > > > >
> > > > > > > On another note, I have a PR for the dynamic broker
> configuration
> > > fix
> > > > > > > here: https://github.com/apache/kafka/pull/11141
> > > > > > >
> > > > > > > Best,
> > > > > > > Ryan Dielhenn
> > > > > > >
> > > > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > > > > > > <ko...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> Please find below the updated release plan for the Apache
> Kafka
> > > > 3.0.0
> > > > > > >> release.
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > > > > >>
> > > > > > >> New suggested dates for the release are as follows:
> > > > > > >>
> > > > > > >> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > > > > >> Feature Freeze is 30 June 2021 (new date, extended by two
> weeks)
> > > > > > >> Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > > > > > >>
> > > > > > >> At least two weeks of stabilization will follow Code Freeze.
> > > > > > >>
> > > > > > >> The release plan is up to date and currently includes all the
> > > > approved
> > > > > > >> KIPs
> > > > > > >> that are targeting 3.0.0.
> > > > > > >>
> > > > > > >> Please let me know if you have any objections with the recent
> > > > > extension
> > > > > > of
> > > > > > >> Feature Freeze and Code Freeze or any other concerns.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Konstantine
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
>
> --
> -Jose
>
Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates
Posted by José Armando García Sancio <js...@confluent.io.INVALID>.
Hey all,
For the KIP-500 work for 3.0 we would like to propose the following
Jiras as blockers:
1. https://issues.apache.org/jira/browse/KAFKA-13168
2. https://issues.apache.org/jira/browse/KAFKA-13165
3. https://issues.apache.org/jira/browse/KAFKA-13161
The description for each Jira should have more details.
Thanks,
-Jose
On Tue, Aug 3, 2021 at 12:14 PM Ryan Dielhenn
<rd...@confluent.io.invalid> wrote:
>
> Hi Konstantine,
>
> I would like to report another bug in KRaft.
>
> The ConfigHandler that processes dynamic broker config deltas in KRaft
> expects that the default resource name for dynamic broker configs is the
> old default entity name used in ZK: "<default>". Since dynamic default
> broker configs are persisted as empty string in the quorum instead of
> "<default>", the brokers are not updating the their default configuration
> when they see empty string as a resource name in the config delta and are
> throwing a NumberFormatException when they try to parse the resource name
> to process it as a per-broker configuration.
>
> I filed a JIRA: https://issues.apache.org/jira/browse/KAFKA-13160
>
> I also have a PR to fix this: https://github.com/apache/kafka/pull/11168
>
> I think that this should be a blocker for 3.0 because dynamic default
> broker configs will not be usable in KRaft otherwise.
>
> Best,
> Ryan Dielhenn
>
> On Sat, Jul 31, 2021 at 10:42 AM Konstantine Karantasis <
> kkarantasis@apache.org> wrote:
>
> > Thanks Ryan,
> >
> > Approved. Seems also like a low risk fix.
> > With that opportunity, let's make sure there are no other configs that
> > would need a similar validation.
> >
> > Konstantine
> >
> > On Fri, Jul 30, 2021 at 8:33 AM Ryan Dielhenn
> > <rd...@confluent.io.invalid> wrote:
> >
> > > Hey Konstantine,
> > >
> > > Thanks for the question. If these configs are not validated the user's
> > > experience will be affected and upgrades from 3.0 will be harder.
> > >
> > > Best,
> > > Ryan Dielhenn
> > >
> > > On Thu, Jul 29, 2021 at 3:59 PM Konstantine Karantasis <
> > > kkarantasis@apache.org> wrote:
> > >
> > > > Thanks for reporting this issue Ryan.
> > > >
> > > > I believe what you mention corresponds to the ticket you created here:
> > > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13151
> > > >
> > > > What happens if the configurations are present but the broker doesn't
> > > fail
> > > > at startup when configured to run in KRaft mode?
> > > > Asking to see if we have any workarounds in our availability.
> > > >
> > > > Thanks,
> > > > Konstantine
> > > >
> > > > On Thu, Jul 29, 2021 at 2:51 PM Ryan Dielhenn
> > > > <rd...@confluent.io.invalid> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Disregard log.clean.policy being included in this blocker.
> > > > >
> > > > > Best,
> > > > > Ryan Dielhenn
> > > > >
> > > > > On Thu, Jul 29, 2021 at 2:38 PM Ryan Dielhenn <
> > rdielhenn@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hey Konstantine,
> > > > > >
> > > > > > I'd like to report another bug in KRaft.
> > > > > >
> > > > > > log.cleanup.policy, alter.config.policy.class.name, and
> > > > > > create.topic.policy.class.name are all unsupported by KRaft but
> > > KRaft
> > > > > > servers allow them to be configured. I believe this should be
> > > > considered
> > > > > a
> > > > > > blocker and that KRaft servers should fail startup if any of these
> > > are
> > > > > > configured. I do not have a PR yet but will soon.
> > > > > >
> > > > > > On another note, I have a PR for the dynamic broker configuration
> > fix
> > > > > > here: https://github.com/apache/kafka/pull/11141
> > > > > >
> > > > > > Best,
> > > > > > Ryan Dielhenn
> > > > > >
> > > > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > > > > > <ko...@confluent.io.invalid> wrote:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> Please find below the updated release plan for the Apache Kafka
> > > 3.0.0
> > > > > >> release.
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > > > >>
> > > > > >> New suggested dates for the release are as follows:
> > > > > >>
> > > > > >> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > > > >> Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > > > > >> Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > > > > >>
> > > > > >> At least two weeks of stabilization will follow Code Freeze.
> > > > > >>
> > > > > >> The release plan is up to date and currently includes all the
> > > approved
> > > > > >> KIPs
> > > > > >> that are targeting 3.0.0.
> > > > > >>
> > > > > >> Please let me know if you have any objections with the recent
> > > > extension
> > > > > of
> > > > > >> Feature Freeze and Code Freeze or any other concerns.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Konstantine
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
--
-Jose