Kafka connector releases

[Arvid Heise <ar...@apache.org>]

Dear Flink devs,

I'd like to initiate three(!) Kafka connector releases. The main reason for
having three releases is that we have been slacking a bit in keeping up
with the latest changes.

Here is the summary:
1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
- Incorporates lots of deprecations for Flink 2 including everything that
is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
(SourceFunction), and KafkaShuffle
- Lots of bugfixes that are very relevant for 1.19 users (and probably work
with older releases)

2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
- Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
incorrectly says that it's avail in 1.19)
- We have discussed some alternatives to this release in [2] but basically
having a separate release is the cleanest solution.
- I'd like to linearize the releases to avoid having to do back or even
forward ports

3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
- Much requested to get the connector out asap for the preview. (I think
the old jar using the removed interfaces should still work)
- Remove all deprecated things
- General spring cleaning (trying to get rid of arch unit violations,
migrate to JUnit5)
- Should we relocate the TableAPI stuff to o.a.f.connectors?

I'd appreciate any feedback and volunteers for RM ;) If you have pending
PRs that should be part of any of those releases, please also write them.

Best,

Arvid

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
[1]
https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878

Re: Kafka connector releases

[Arvid Heise <ah...@confluent.io.INVALID>]

Hi Yanquan,

thank you very much for taking care of this.

I propose to merge your PR without E2E into 4.0-SNAPSHOT. Then we wait
for Flink 2.0.0 with the fix, reenable E2E and lastly we can cut an
official 4.0.0 release (we need to wait for 2.0.0 anyways for the
release).
We could release a 4.0.0-preview if needed (but 4.0-SNAPSHOT could be
good enough).

In this way, we can merge your PR and you don't need to rebase further
while also not sacrificing any reliability.

WDYT?

Best,

Arvid

On Thu, Feb 13, 2025 at 1:45 PM Yanquan Lv <de...@gmail.com> wrote:
>
> Hi, Devs.
>
> I am now working on the Kafka connector bump to Flink 2.0 in 4.0.0 version, and I have already opened a PR for this. Since there is a bug that fix in [FLINK-36568](Flink 2.0) in Flink 2.0-preview1 that would cause fail to run e2e case, so I comment out these e2e tests to pass CI.
> However, considering the reliability of the release, the community is concerned about the lack of e2e testing, So the work on bump 2.0-preview1 has not been completed yet.
>
> At present, the official version of Flink 2.0 is also about to be released. would it be better for us to directly bump to this official version before continuing with the release of flink-kafka-connector 4.0.0?
> Thanks for any feedback from the community regarding the release of version 4.0.0.
>
>
> [1] https://github.com/apache/flink-connector-kafka/pull/140
> [2] https://issues.apache.org/jira/browse/FLINK-36568
>
>
>
> > 2024年11月15日 04:09，Arvid Heise <ar...@apache.org> 写道：
> >
> > Hi all,
> >
> > I just sent out the first release candidate for 3.4.0. The only difference
> > to 3.3.0 is that we cease to support 1.19 in favor of supporting the open
> > lineage interfaces [1] that have been added to Flink 1.20.
> >
> > With that release under way (please verify and vote!), the main branch is
> > now good to go for the Flink 2.0-preview release. @Qingsheng Ren
> > <re...@gmail.com> and Leonard are driving that release, Yanquan already
> > volunteered to help. I'm pretty sure more help is always welcome. Maybe it
> > even makes sense to organize in Slack for that release.
> >
> > Best of luck and don't hesitate to reach out to me if you need another
> > opinion.
> >
> > Arvid
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-34466
> >
> > On Wed, Nov 6, 2024 at 2:16 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> >> Thanks @Yanquan for track this thread, and thanks @Arvid for the
> >> information, it makes sense to me
> >>
> >> Qingsheng will drive the release of flink-2.0-preview of Flink Kafka
> >> connector, and I’d like assist it too.
> >>
> >> Best,
> >> Leonard
> >>
> >>
> >>> 2024年11月6日 下午6:58，Arvid Heise <ah...@confluent.io.INVALID> 写道：
> >>>
> >>> Hi Yanquan,
> >>>
> >>> the current state of the 3.4.0 release is that it's still pending on the
> >>> lineage PR [1] which I expect to be merged next week (the author is on
> >>> vacation). The release cut would then happen right afterwards.
> >>>
> >>> After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink
> >>> 2.0-preview. @Qingsheng Ren <re...@gmail.com> and Leonard wanted to
> >>> drive that release. I already prepared a bit by thoroughly annotating
> >>> everything with Deprecated but the whole test side needs a bigger
> >> cleanup.
> >>> It's probably also a good time to bump other dependencies.
> >>> Could you please sync with the two release managers? At least Qingsheng
> >> is
> >>> responsive in the Flink slack - I talked to him quite a bit there.
> >>>
> >>> If there is a pressing need to start earlier, we can also cut the 3.4
> >>> branch (which is then effectively the 3.3 branch) earlier and backport
> >> the
> >>> lineage PR (it's just one commit ultimately). I'd leave that decision to
> >>> the two release managers for 4.0.0 mentioned before.
> >>>
> >>> One thing to note for 4.0.0 is that we need to solve the transaction
> >>> management issues with the KafkaSink [2]. It's blocking larger users from
> >>> adopting the KafkaSink which will be the only option for Flink 2.0. I
> >> have
> >>> started designing a solution.
> >>>
> >>> Best,
> >>>
> >>> Arvid
> >>>
> >>>
> >>> [1] https://github.com/apache/flink-connector-kafka/pull/130
> >>> [2] https://issues.apache.org/jira/browse/FLINK-34554
> >>>
> >>> On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <de...@gmail.com> wrote:
> >>>
> >>>> Hi, Arvid.
> >>>>
> >>>> It has been a month and we are glad to see that we have completed the
> >>>> release of Kafka 3.3.0 targeting 1.19 and 1.20.
> >>>>
> >>>> Considering that Flink 2.0-preview1 has already been released, I would
> >>>> like to know about our plans and progress for bumping to 2.0-preview1.
> >>>> I tested the changes required for bump to 2.0-preview1 locally and found
> >>>> that the adaptation changes made in the production code based on
> >>>> FlinkKafkaProducer Depreciated work were relatively clear and the
> >> amount of
> >>>> change was not significant. However, the headache was that there were
> >> many
> >>>> adjustments needed in the test code.
> >>>>
> >>>> I would like to know if there is already work in the community to bump
> >> to
> >>>> 2.0-preview1. If not, I can help complete this task (but some
> >> suggestions
> >>>> may be needed for testing the adaptation in the code).
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
> >>>>>
> >>>>> Dear Flink devs,
> >>>>>
> >>>>> I'd like to initiate three(!) Kafka connector releases. The main reason
> >>>> for
> >>>>> having three releases is that we have been slacking a bit in keeping up
> >>>>> with the latest changes.
> >>>>>
> >>>>> Here is the summary:
> >>>>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> >>>>> - Incorporates lots of deprecations for Flink 2 including everything
> >> that
> >>>>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> >>>>> (SourceFunction), and KafkaShuffle
> >>>>> - Lots of bugfixes that are very relevant for 1.19 users (and probably
> >>>> work
> >>>>> with older releases)
> >>>>>
> >>>>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> >>>>> - Incorporates lineage tracing which is only available in 1.20 [1]
> >> (FLIP
> >>>>> incorrectly says that it's avail in 1.19)
> >>>>> - We have discussed some alternatives to this release in [2] but
> >>>> basically
> >>>>> having a separate release is the cleanest solution.
> >>>>> - I'd like to linearize the releases to avoid having to do back or even
> >>>>> forward ports
> >>>>>
> >>>>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> >>>>> - Much requested to get the connector out asap for the preview. (I
> >> think
> >>>>> the old jar using the removed interfaces should still work)
> >>>>> - Remove all deprecated things
> >>>>> - General spring cleaning (trying to get rid of arch unit violations,
> >>>>> migrate to JUnit5)
> >>>>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> >>>>>
> >>>>> I'd appreciate any feedback and volunteers for RM ;) If you have
> >> pending
> >>>>> PRs that should be part of any of those releases, please also write
> >> them.
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> Arvid
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> >>>>> [1]
> >>>>>
> >>>>
> >> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> >>>>
> >>>>
> >>
> >>
>

Re: Kafka connector releases

[Yanquan Lv <de...@gmail.com>]

Hi, Devs.

I am now working on the Kafka connector bump to Flink 2.0 in 4.0.0 version, and I have already opened a PR for this. Since there is a bug that fix in [FLINK-36568](Flink 2.0) in Flink 2.0-preview1 that would cause fail to run e2e case, so I comment out these e2e tests to pass CI. 
However, considering the reliability of the release, the community is concerned about the lack of e2e testing, So the work on bump 2.0-preview1 has not been completed yet.

At present, the official version of Flink 2.0 is also about to be released. would it be better for us to directly bump to this official version before continuing with the release of flink-kafka-connector 4.0.0?
Thanks for any feedback from the community regarding the release of version 4.0.0.


[1] https://github.com/apache/flink-connector-kafka/pull/140
[2] https://issues.apache.org/jira/browse/FLINK-36568



> 2024年11月15日 04:09，Arvid Heise <ar...@apache.org> 写道：
> 
> Hi all,
> 
> I just sent out the first release candidate for 3.4.0. The only difference
> to 3.3.0 is that we cease to support 1.19 in favor of supporting the open
> lineage interfaces [1] that have been added to Flink 1.20.
> 
> With that release under way (please verify and vote!), the main branch is
> now good to go for the Flink 2.0-preview release. @Qingsheng Ren
> <re...@gmail.com> and Leonard are driving that release, Yanquan already
> volunteered to help. I'm pretty sure more help is always welcome. Maybe it
> even makes sense to organize in Slack for that release.
> 
> Best of luck and don't hesitate to reach out to me if you need another
> opinion.
> 
> Arvid
> 
> [1] https://issues.apache.org/jira/browse/FLINK-34466
> 
> On Wed, Nov 6, 2024 at 2:16 PM Leonard Xu <xb...@gmail.com> wrote:
> 
>> Thanks @Yanquan for track this thread, and thanks @Arvid for the
>> information, it makes sense to me
>> 
>> Qingsheng will drive the release of flink-2.0-preview of Flink Kafka
>> connector, and I’d like assist it too.
>> 
>> Best,
>> Leonard
>> 
>> 
>>> 2024年11月6日 下午6:58，Arvid Heise <ah...@confluent.io.INVALID> 写道：
>>> 
>>> Hi Yanquan,
>>> 
>>> the current state of the 3.4.0 release is that it's still pending on the
>>> lineage PR [1] which I expect to be merged next week (the author is on
>>> vacation). The release cut would then happen right afterwards.
>>> 
>>> After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink
>>> 2.0-preview. @Qingsheng Ren <re...@gmail.com> and Leonard wanted to
>>> drive that release. I already prepared a bit by thoroughly annotating
>>> everything with Deprecated but the whole test side needs a bigger
>> cleanup.
>>> It's probably also a good time to bump other dependencies.
>>> Could you please sync with the two release managers? At least Qingsheng
>> is
>>> responsive in the Flink slack - I talked to him quite a bit there.
>>> 
>>> If there is a pressing need to start earlier, we can also cut the 3.4
>>> branch (which is then effectively the 3.3 branch) earlier and backport
>> the
>>> lineage PR (it's just one commit ultimately). I'd leave that decision to
>>> the two release managers for 4.0.0 mentioned before.
>>> 
>>> One thing to note for 4.0.0 is that we need to solve the transaction
>>> management issues with the KafkaSink [2]. It's blocking larger users from
>>> adopting the KafkaSink which will be the only option for Flink 2.0. I
>> have
>>> started designing a solution.
>>> 
>>> Best,
>>> 
>>> Arvid
>>> 
>>> 
>>> [1] https://github.com/apache/flink-connector-kafka/pull/130
>>> [2] https://issues.apache.org/jira/browse/FLINK-34554
>>> 
>>> On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <de...@gmail.com> wrote:
>>> 
>>>> Hi, Arvid.
>>>> 
>>>> It has been a month and we are glad to see that we have completed the
>>>> release of Kafka 3.3.0 targeting 1.19 and 1.20.
>>>> 
>>>> Considering that Flink 2.0-preview1 has already been released, I would
>>>> like to know about our plans and progress for bumping to 2.0-preview1.
>>>> I tested the changes required for bump to 2.0-preview1 locally and found
>>>> that the adaptation changes made in the production code based on
>>>> FlinkKafkaProducer Depreciated work were relatively clear and the
>> amount of
>>>> change was not significant. However, the headache was that there were
>> many
>>>> adjustments needed in the test code.
>>>> 
>>>> I would like to know if there is already work in the community to bump
>> to
>>>> 2.0-preview1. If not, I can help complete this task (but some
>> suggestions
>>>> may be needed for testing the adaptation in the code).
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
>>>>> 
>>>>> Dear Flink devs,
>>>>> 
>>>>> I'd like to initiate three(!) Kafka connector releases. The main reason
>>>> for
>>>>> having three releases is that we have been slacking a bit in keeping up
>>>>> with the latest changes.
>>>>> 
>>>>> Here is the summary:
>>>>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
>>>>> - Incorporates lots of deprecations for Flink 2 including everything
>> that
>>>>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
>>>>> (SourceFunction), and KafkaShuffle
>>>>> - Lots of bugfixes that are very relevant for 1.19 users (and probably
>>>> work
>>>>> with older releases)
>>>>> 
>>>>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
>>>>> - Incorporates lineage tracing which is only available in 1.20 [1]
>> (FLIP
>>>>> incorrectly says that it's avail in 1.19)
>>>>> - We have discussed some alternatives to this release in [2] but
>>>> basically
>>>>> having a separate release is the cleanest solution.
>>>>> - I'd like to linearize the releases to avoid having to do back or even
>>>>> forward ports
>>>>> 
>>>>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
>>>>> - Much requested to get the connector out asap for the preview. (I
>> think
>>>>> the old jar using the removed interfaces should still work)
>>>>> - Remove all deprecated things
>>>>> - General spring cleaning (trying to get rid of arch unit violations,
>>>>> migrate to JUnit5)
>>>>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>>>>> 
>>>>> I'd appreciate any feedback and volunteers for RM ;) If you have
>> pending
>>>>> PRs that should be part of any of those releases, please also write
>> them.
>>>>> 
>>>>> Best,
>>>>> 
>>>>> Arvid
>>>>> 
>>>>> [1]
>>>>> 
>>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
>>>>> [1]
>>>>> 
>>>> 
>> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>>>> 
>>>> 
>> 
>> 

Re: Kafka connector releases

[Arvid Heise <ar...@apache.org>]

Hi all,

I just sent out the first release candidate for 3.4.0. The only difference
to 3.3.0 is that we cease to support 1.19 in favor of supporting the open
lineage interfaces [1] that have been added to Flink 1.20.

With that release under way (please verify and vote!), the main branch is
now good to go for the Flink 2.0-preview release. @Qingsheng Ren
<re...@gmail.com> and Leonard are driving that release, Yanquan already
volunteered to help. I'm pretty sure more help is always welcome. Maybe it
even makes sense to organize in Slack for that release.

Best of luck and don't hesitate to reach out to me if you need another
opinion.

Arvid

[1] https://issues.apache.org/jira/browse/FLINK-34466

On Wed, Nov 6, 2024 at 2:16 PM Leonard Xu <xb...@gmail.com> wrote:

> Thanks @Yanquan for track this thread, and thanks @Arvid for the
> information, it makes sense to me
>
> Qingsheng will drive the release of flink-2.0-preview of Flink Kafka
> connector, and I’d like assist it too.
>
> Best,
> Leonard
>
>
> > 2024年11月6日 下午6:58，Arvid Heise <ah...@confluent.io.INVALID> 写道：
> >
> > Hi Yanquan,
> >
> > the current state of the 3.4.0 release is that it's still pending on the
> > lineage PR [1] which I expect to be merged next week (the author is on
> > vacation). The release cut would then happen right afterwards.
> >
> > After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink
> > 2.0-preview. @Qingsheng Ren <re...@gmail.com> and Leonard wanted to
> > drive that release. I already prepared a bit by thoroughly annotating
> > everything with Deprecated but the whole test side needs a bigger
> cleanup.
> > It's probably also a good time to bump other dependencies.
> > Could you please sync with the two release managers? At least Qingsheng
> is
> > responsive in the Flink slack - I talked to him quite a bit there.
> >
> > If there is a pressing need to start earlier, we can also cut the 3.4
> > branch (which is then effectively the 3.3 branch) earlier and backport
> the
> > lineage PR (it's just one commit ultimately). I'd leave that decision to
> > the two release managers for 4.0.0 mentioned before.
> >
> > One thing to note for 4.0.0 is that we need to solve the transaction
> > management issues with the KafkaSink [2]. It's blocking larger users from
> > adopting the KafkaSink which will be the only option for Flink 2.0. I
> have
> > started designing a solution.
> >
> > Best,
> >
> > Arvid
> >
> >
> > [1] https://github.com/apache/flink-connector-kafka/pull/130
> > [2] https://issues.apache.org/jira/browse/FLINK-34554
> >
> > On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <de...@gmail.com> wrote:
> >
> >> Hi, Arvid.
> >>
> >> It has been a month and we are glad to see that we have completed the
> >> release of Kafka 3.3.0 targeting 1.19 and 1.20.
> >>
> >> Considering that Flink 2.0-preview1 has already been released, I would
> >> like to know about our plans and progress for bumping to 2.0-preview1.
> >> I tested the changes required for bump to 2.0-preview1 locally and found
> >> that the adaptation changes made in the production code based on
> >> FlinkKafkaProducer Depreciated work were relatively clear and the
> amount of
> >> change was not significant. However, the headache was that there were
> many
> >> adjustments needed in the test code.
> >>
> >> I would like to know if there is already work in the community to bump
> to
> >> 2.0-preview1. If not, I can help complete this task (but some
> suggestions
> >> may be needed for testing the adaptation in the code).
> >>
> >>
> >>
> >>
> >>
> >>> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
> >>>
> >>> Dear Flink devs,
> >>>
> >>> I'd like to initiate three(!) Kafka connector releases. The main reason
> >> for
> >>> having three releases is that we have been slacking a bit in keeping up
> >>> with the latest changes.
> >>>
> >>> Here is the summary:
> >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> >>> - Incorporates lots of deprecations for Flink 2 including everything
> that
> >>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> >>> (SourceFunction), and KafkaShuffle
> >>> - Lots of bugfixes that are very relevant for 1.19 users (and probably
> >> work
> >>> with older releases)
> >>>
> >>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> >>> - Incorporates lineage tracing which is only available in 1.20 [1]
> (FLIP
> >>> incorrectly says that it's avail in 1.19)
> >>> - We have discussed some alternatives to this release in [2] but
> >> basically
> >>> having a separate release is the cleanest solution.
> >>> - I'd like to linearize the releases to avoid having to do back or even
> >>> forward ports
> >>>
> >>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> >>> - Much requested to get the connector out asap for the preview. (I
> think
> >>> the old jar using the removed interfaces should still work)
> >>> - Remove all deprecated things
> >>> - General spring cleaning (trying to get rid of arch unit violations,
> >>> migrate to JUnit5)
> >>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> >>>
> >>> I'd appreciate any feedback and volunteers for RM ;) If you have
> pending
> >>> PRs that should be part of any of those releases, please also write
> them.
> >>>
> >>> Best,
> >>>
> >>> Arvid
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> >>> [1]
> >>>
> >>
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> >>
> >>
>
>

Re: Kafka connector releases

[Leonard Xu <xb...@gmail.com>]

Thanks @Yanquan for track this thread, and thanks @Arvid for the information, it makes sense to me

Qingsheng will drive the release of flink-2.0-preview of Flink Kafka connector, and I’d like assist it too.

Best,
Leonard


> 2024年11月6日 下午6:58，Arvid Heise <ah...@confluent.io.INVALID> 写道：
> 
> Hi Yanquan,
> 
> the current state of the 3.4.0 release is that it's still pending on the
> lineage PR [1] which I expect to be merged next week (the author is on
> vacation). The release cut would then happen right afterwards.
> 
> After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink
> 2.0-preview. @Qingsheng Ren <re...@gmail.com> and Leonard wanted to
> drive that release. I already prepared a bit by thoroughly annotating
> everything with Deprecated but the whole test side needs a bigger cleanup.
> It's probably also a good time to bump other dependencies.
> Could you please sync with the two release managers? At least Qingsheng is
> responsive in the Flink slack - I talked to him quite a bit there.
> 
> If there is a pressing need to start earlier, we can also cut the 3.4
> branch (which is then effectively the 3.3 branch) earlier and backport the
> lineage PR (it's just one commit ultimately). I'd leave that decision to
> the two release managers for 4.0.0 mentioned before.
> 
> One thing to note for 4.0.0 is that we need to solve the transaction
> management issues with the KafkaSink [2]. It's blocking larger users from
> adopting the KafkaSink which will be the only option for Flink 2.0. I have
> started designing a solution.
> 
> Best,
> 
> Arvid
> 
> 
> [1] https://github.com/apache/flink-connector-kafka/pull/130
> [2] https://issues.apache.org/jira/browse/FLINK-34554
> 
> On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <de...@gmail.com> wrote:
> 
>> Hi, Arvid.
>> 
>> It has been a month and we are glad to see that we have completed the
>> release of Kafka 3.3.0 targeting 1.19 and 1.20.
>> 
>> Considering that Flink 2.0-preview1 has already been released, I would
>> like to know about our plans and progress for bumping to 2.0-preview1.
>> I tested the changes required for bump to 2.0-preview1 locally and found
>> that the adaptation changes made in the production code based on
>> FlinkKafkaProducer Depreciated work were relatively clear and the amount of
>> change was not significant. However, the headache was that there were many
>> adjustments needed in the test code.
>> 
>> I would like to know if there is already work in the community to bump to
>> 2.0-preview1. If not, I can help complete this task (but some suggestions
>> may be needed for testing the adaptation in the code).
>> 
>> 
>> 
>> 
>> 
>>> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
>>> 
>>> Dear Flink devs,
>>> 
>>> I'd like to initiate three(!) Kafka connector releases. The main reason
>> for
>>> having three releases is that we have been slacking a bit in keeping up
>>> with the latest changes.
>>> 
>>> Here is the summary:
>>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
>>> - Incorporates lots of deprecations for Flink 2 including everything that
>>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
>>> (SourceFunction), and KafkaShuffle
>>> - Lots of bugfixes that are very relevant for 1.19 users (and probably
>> work
>>> with older releases)
>>> 
>>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
>>> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
>>> incorrectly says that it's avail in 1.19)
>>> - We have discussed some alternatives to this release in [2] but
>> basically
>>> having a separate release is the cleanest solution.
>>> - I'd like to linearize the releases to avoid having to do back or even
>>> forward ports
>>> 
>>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
>>> - Much requested to get the connector out asap for the preview. (I think
>>> the old jar using the removed interfaces should still work)
>>> - Remove all deprecated things
>>> - General spring cleaning (trying to get rid of arch unit violations,
>>> migrate to JUnit5)
>>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>>> 
>>> I'd appreciate any feedback and volunteers for RM ;) If you have pending
>>> PRs that should be part of any of those releases, please also write them.
>>> 
>>> Best,
>>> 
>>> Arvid
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
>>> [1]
>>> 
>> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>> 
>> 

Re: Kafka connector releases

[Arvid Heise <ah...@confluent.io.INVALID>]

Hi Yanquan,

the current state of the 3.4.0 release is that it's still pending on the
lineage PR [1] which I expect to be merged next week (the author is on
vacation). The release cut would then happen right afterwards.

After the release cut, we can then bump to 4.0.0-SNAPSHOT and Flink
2.0-preview. @Qingsheng Ren <re...@gmail.com> and Leonard wanted to
drive that release. I already prepared a bit by thoroughly annotating
everything with Deprecated but the whole test side needs a bigger cleanup.
It's probably also a good time to bump other dependencies.
Could you please sync with the two release managers? At least Qingsheng is
responsive in the Flink slack - I talked to him quite a bit there.

If there is a pressing need to start earlier, we can also cut the 3.4
branch (which is then effectively the 3.3 branch) earlier and backport the
lineage PR (it's just one commit ultimately). I'd leave that decision to
the two release managers for 4.0.0 mentioned before.

One thing to note for 4.0.0 is that we need to solve the transaction
management issues with the KafkaSink [2]. It's blocking larger users from
adopting the KafkaSink which will be the only option for Flink 2.0. I have
started designing a solution.

Best,

Arvid


[1] https://github.com/apache/flink-connector-kafka/pull/130
[2] https://issues.apache.org/jira/browse/FLINK-34554

On Tue, Nov 5, 2024 at 4:48 AM Yanquan Lv <de...@gmail.com> wrote:

> Hi, Arvid.
>
> It has been a month and we are glad to see that we have completed the
> release of Kafka 3.3.0 targeting 1.19 and 1.20.
>
> Considering that Flink 2.0-preview1 has already been released, I would
> like to know about our plans and progress for bumping to 2.0-preview1.
> I tested the changes required for bump to 2.0-preview1 locally and found
> that the adaptation changes made in the production code based on
> FlinkKafkaProducer Depreciated work were relatively clear and the amount of
> change was not significant. However, the headache was that there were many
> adjustments needed in the test code.
>
> I would like to know if there is already work in the community to bump to
> 2.0-preview1. If not, I can help complete this task (but some suggestions
> may be needed for testing the adaptation in the code).
>
>
>
>
>
> > 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
> >
> > Dear Flink devs,
> >
> > I'd like to initiate three(!) Kafka connector releases. The main reason
> for
> > having three releases is that we have been slacking a bit in keeping up
> > with the latest changes.
> >
> > Here is the summary:
> > 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> > - Incorporates lots of deprecations for Flink 2 including everything that
> > is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> > (SourceFunction), and KafkaShuffle
> > - Lots of bugfixes that are very relevant for 1.19 users (and probably
> work
> > with older releases)
> >
> > 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> > - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> > incorrectly says that it's avail in 1.19)
> > - We have discussed some alternatives to this release in [2] but
> basically
> > having a separate release is the cleanest solution.
> > - I'd like to linearize the releases to avoid having to do back or even
> > forward ports
> >
> > 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> > - Much requested to get the connector out asap for the preview. (I think
> > the old jar using the removed interfaces should still work)
> > - Remove all deprecated things
> > - General spring cleaning (trying to get rid of arch unit violations,
> > migrate to JUnit5)
> > - Should we relocate the TableAPI stuff to o.a.f.connectors?
> >
> > I'd appreciate any feedback and volunteers for RM ;) If you have pending
> > PRs that should be part of any of those releases, please also write them.
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > [1]
> >
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>
>

Re: Kafka connector releases

[Yanquan Lv <de...@gmail.com>]

Hi, Arvid.

It has been a month and we are glad to see that we have completed the release of Kafka 3.3.0 targeting 1.19 and 1.20.

Considering that Flink 2.0-preview1 has already been released, I would like to know about our plans and progress for bumping to 2.0-preview1.
I tested the changes required for bump to 2.0-preview1 locally and found that the adaptation changes made in the production code based on FlinkKafkaProducer Depreciated work were relatively clear and the amount of change was not significant. However, the headache was that there were many adjustments needed in the test code.

I would like to know if there is already work in the community to bump to 2.0-preview1. If not, I can help complete this task (but some suggestions may be needed for testing the adaptation in the code).





> 2024年9月27日 16:23，Arvid Heise <ar...@apache.org> 写道：
> 
> Dear Flink devs,
> 
> I'd like to initiate three(!) Kafka connector releases. The main reason for
> having three releases is that we have been slacking a bit in keeping up
> with the latest changes.
> 
> Here is the summary:
> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> - Incorporates lots of deprecations for Flink 2 including everything that
> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> (SourceFunction), and KafkaShuffle
> - Lots of bugfixes that are very relevant for 1.19 users (and probably work
> with older releases)
> 
> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> incorrectly says that it's avail in 1.19)
> - We have discussed some alternatives to this release in [2] but basically
> having a separate release is the cleanest solution.
> - I'd like to linearize the releases to avoid having to do back or even
> forward ports
> 
> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> - Much requested to get the connector out asap for the preview. (I think
> the old jar using the removed interfaces should still work)
> - Remove all deprecated things
> - General spring cleaning (trying to get rid of arch unit violations,
> migrate to JUnit5)
> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> 
> I'd appreciate any feedback and volunteers for RM ;) If you have pending
> PRs that should be part of any of those releases, please also write them.
> 
> Best,
> 
> Arvid
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> [1]
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878

Re: Kafka connector releases

[Arvid Heise <ar...@apache.org>]

Hi Danny,

thanks for bringing this to my attention. I'm also in favor of starting
with Flink 2.0. It's just a nice opportunity to break with some traditions
and we are almost there anyways. We could resume the discussion over there
if you'd like.

Regarding the deprecation. I think you misunderstood me. I was talking
about deprecating the KafkaShuffle and all the old API around
Consumer/Producer (which we already deprecated in 1.14). [1]
But I'd also deprecate 1.18 support with that release, so all that you say
is additionally true.

Since we are on the topic of deprecation and I got your attention: Does
anyone have strong opinions about moving the TableAPI stuff into the
o.a.f.connectors package? It would probably moving things + provide stubs
with deprecation in the old location that extend from the actual things
similar to the FlinkKafkaPartitioner in [1]. We can start a separate
discussion and I don't have a strong opinion since I'm not a TableAPI/SQL
guy. I'm also getting low on OSS contribution time.

Best,

Arvid

[1] https://github.com/apache/flink-connector-kafka/pull/121

On Mon, Sep 30, 2024 at 4:49 PM Danny Cranmer <da...@apache.org>
wrote:

> Thanks for driving this Arvid.
>
> +1 for the releases and the RM
>
> For the 4.0.0 version it would be good to drop the Flink version from the
> connector version (4.0.0, not 4.0.0-2.0). I started a discussion some time
> ago [1], however the thread went stale. But I think the feedback was
> generally aligned. I would propose we release 4.0.0 (or 4.0.0-preview?) and
> document this only supports Flink 2.0-preview.
>
> > I don't think we have any precedent of deprecating API in a bugfix
> release.
>
> I have not seen us do that before for connectors. I would personally prefer
> a 3.3.0 if we are dropping Flink 1.18 support, although I agree it is a
> grey area in the guidelines. This allows us to more easily create a
> 3.2.1-1.18 hotfix later incase the need arises and the community agrees to
> push another 1.18 hotfix.
>
> >  It's not obvious to me if a bugfix release counts towards our
> deprecation policy where we have one release between deprecation and
> removal.
>
> Given that we encode the Flink version in the connector version, it is
> difficult/impossible to not notice it has been deprecated. Since the user
> would bump the version from 3.2.0-1.18 to 3.3.0-1.18, and this artifact
> would not be found in Maven central, and fail the build.
>
> > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If they
> are fully compatible, what about not dropping support for 1.18 in 3.2.1,
> and we release one more version 3.2.1-1.20? Then we can use 3.3.0 for the
> new lineage feature in 1.20 and drop support for 1.18 and 1.19.
>
> This is a possibility, but still results in 3 releases, so only worth it if
> it is simpler
>
> Thanks,
> Danny
>
> [1] https://lists.apache.org/thread/byy8fgkr8dbrrdv2nxpxkrz18h904r9b
>
>
> On Mon, Sep 30, 2024 at 10:36 AM Arvid Heise <ar...@apache.org> wrote:
>
> > Hi folks,
> >
> > thanks for the feedback. I'm still not convinced that it's the best
> > approach to start with a bugfix release but I did a poor job in
> > highlighting my thoughts.
> > * I don't think we have any precedent of deprecating API in a bugfix
> > release.
> > * It's not obvious to me if a bugfix release counts towards our
> deprecation
> > policy where we have one release between deprecation and removal. A user
> > could use 3.2.0 not notice the 3.2.1 and see the deprecation only in
> 3.3.0.
> > Then the user could be surprised by the removal in 4.0.0. (I'd personally
> > see our policy too strict regarding major releases but that's what we did
> > for Flink 2)
> > * For the deprecation, I also introduced a new interface KafkaPartitioner
> > that's a drop-in replacement for FlinkKafkaPartitioner. This is new API
> and
> > should be part of a new minor release.
> > * Finally, there is one new feature: [FLINK-22748] Allow dynamic target
> > topic selection in SQL Kafka sinks [1].
> > * I'd drop 1.18 still and support 1.20 instead.
> > * I'd also bump kafka-clients to 3.7.1
> >
> > WDYT?
> >
> > Best,
> >
> > Arvid
> >
> > [1] https://github.com/apache/flink-connector-kafka/pull/109
> >
> > On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > Thanks Arvid for the volunteering!
> > >
> > > + 1 for all of the three releases and RM candidate.Qingsheng and I
> would
> > > like to help the 4.0.0-preview which follows the Flink 2.0 preview,
> > > please feel free to ping us if you need any help.
> > >
> > > Btw, for other external connectors which  highest supported flink
> version
> > > still is 1.17 or 1.18,  I also hope we can have dedicated plan to bump
> > > their version ASAP, we can
> > > start a new thread to track other external connector releases.
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > > > 2024年9月27日 下午6:54，Qingsheng Ren <re...@apache.org> 写道：
> > > >
> > > > Thanks for the work, Arvid!
> > > >
> > > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If
> > > > they are fully compatible, what about not dropping support for 1.18
> in
> > > > 3.2.1, and we release one more version 3.2.1-1.20? Then we can use
> > > > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18
> > > > and 1.19.
> > > >
> > > > And for the 4.0.0-preview version I'd like to help with it :-)
> > > >
> > > > Best,
> > > > Qingsheng
> > > >
> > > > On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise <ar...@apache.org>
> wrote:
> > > >>
> > > >> Hi David,
> > > >>
> > > >> thank you very much for your reply.
> > > >>
> > > >>> Some thoughts on whether we need the 3 deliverables. And whether we
> > > could
> > > >> follow more traditional fixpack numbering:
> > > >>> I see that there is already a release for 1.19
> > > >>
> > >
> >
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> > > >> .
> > > >> I am wondering why we need the first deliverable. If we need it for
> > the
> > > bug
> > > >> fixes , why not have a 3.2.1?
> > > >> I forgot the most important part of the first release: drop Flink
> 1.18
> > > >> support and add Flink 1.20 support. Hence I wouldn't want to mix
> that
> > > into
> > > >> a bugfix release. I think this is in line with the previous minor
> > > releases.
> > > >>
> > > >>> I assume that kafka-3.4.0 will not work with previous Flink
> releases.
> > > >> Would it be worth have a config switch to enable the lineage in the
> > > >> connector so that we could use it with 1.19?  We could maybe do a
> 3.3
> > if
> > > >> this was the case.
> > > >> Yes, as outlined in the discussion linked in the original message,
> we
> > > need
> > > >> to mixin new interfaces. Afaik classloading will fail if the
> > interfaces
> > > are
> > > >> not present, even if the methods are not used. So I don't see how we
> > can
> > > >> use a config switch to make it happen (except with code
> duplication).
> > > >> However, I'm grateful for any ideas to avoid this release.
> > > >>
> > > >> Best,
> > > >>
> > > >> Arvid
> > > >>
> > > >> On Fri, Sep 27, 2024 at 11:11 AM David Radley <
> > david_radley@uk.ibm.com>
> > > >> wrote:
> > > >>
> > > >>> Hi Arvid,
> > > >>> Some thoughts on whether we need the 3 deliverables. And whether we
> > > could
> > > >>> follow more traditional fixpack numbering:
> > > >>> I see that there is already a release for 1.19
> > > >>>
> > >
> >
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> > > >>> . I am wondering why we need the first deliverable. If we need it
> for
> > > the
> > > >>> bug fixes , why not have a 3.2.1?
> > > >>> I assume that kafka-3.4.0 will not work with previous Flink
> releases.
> > > >>> Would it be worth have a config switch to enable the lineage in the
> > > >>> connector so that we could use it with 1.19?  We could maybe do a
> 3.3
> > > if
> > > >>> this was the case.
> > > >>>
> > > >>> WDYT?
> > > >>>  Kind regards, David.
> > > >>>
> > > >>>
> > > >>>
> > > >>> From: Arvid Heise <ar...@apache.org>
> > > >>> Date: Friday, 27 September 2024 at 09:24
> > > >>> To: dev@flink.apache.org <de...@flink.apache.org>
> > > >>> Subject: [EXTERNAL] Kafka connector releases
> > > >>> Dear Flink devs,
> > > >>>
> > > >>> I'd like to initiate three(!) Kafka connector releases. The main
> > > reason for
> > > >>> having three releases is that we have been slacking a bit in
> keeping
> > up
> > > >>> with the latest changes.
> > > >>>
> > > >>> Here is the summary:
> > > >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> > > >>> - Incorporates lots of deprecations for Flink 2 including
> everything
> > > that
> > > >>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> > > >>> (SourceFunction), and KafkaShuffle
> > > >>> - Lots of bugfixes that are very relevant for 1.19 users (and
> > probably
> > > work
> > > >>> with older releases)
> > > >>>
> > > >>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> > > >>> - Incorporates lineage tracing which is only available in 1.20 [1]
> > > (FLIP
> > > >>> incorrectly says that it's avail in 1.19)
> > > >>> - We have discussed some alternatives to this release in [2] but
> > > basically
> > > >>> having a separate release is the cleanest solution.
> > > >>> - I'd like to linearize the releases to avoid having to do back or
> > even
> > > >>> forward ports
> > > >>>
> > > >>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> > > >>> - Much requested to get the connector out asap for the preview. (I
> > > think
> > > >>> the old jar using the removed interfaces should still work)
> > > >>> - Remove all deprecated things
> > > >>> - General spring cleaning (trying to get rid of arch unit
> violations,
> > > >>> migrate to JUnit5)
> > > >>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> > > >>>
> > > >>> I'd appreciate any feedback and volunteers for RM ;) If you have
> > > pending
> > > >>> PRs that should be part of any of those releases, please also write
> > > them.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Arvid
> > > >>>
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > >>> [1]
> > > >>>
> > > >>>
> > >
> >
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> > > >>>
> > > >>> Unless otherwise stated above:
> > > >>>
> > > >>> IBM United Kingdom Limited
> > > >>> Registered in England and Wales with number 741598
> > > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> > 3AU
> > > >>>
> > >
> > >
> >
>

Re: Kafka connector releases

[Danny Cranmer <da...@apache.org>]

Thanks for driving this Arvid.

+1 for the releases and the RM

For the 4.0.0 version it would be good to drop the Flink version from the
connector version (4.0.0, not 4.0.0-2.0). I started a discussion some time
ago [1], however the thread went stale. But I think the feedback was
generally aligned. I would propose we release 4.0.0 (or 4.0.0-preview?) and
document this only supports Flink 2.0-preview.

> I don't think we have any precedent of deprecating API in a bugfix
release.

I have not seen us do that before for connectors. I would personally prefer
a 3.3.0 if we are dropping Flink 1.18 support, although I agree it is a
grey area in the guidelines. This allows us to more easily create a
3.2.1-1.18 hotfix later incase the need arises and the community agrees to
push another 1.18 hotfix.

>  It's not obvious to me if a bugfix release counts towards our
deprecation policy where we have one release between deprecation and
removal.

Given that we encode the Flink version in the connector version, it is
difficult/impossible to not notice it has been deprecated. Since the user
would bump the version from 3.2.0-1.18 to 3.3.0-1.18, and this artifact
would not be found in Maven central, and fail the build.

> I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If they
are fully compatible, what about not dropping support for 1.18 in 3.2.1,
and we release one more version 3.2.1-1.20? Then we can use 3.3.0 for the
new lineage feature in 1.20 and drop support for 1.18 and 1.19.

This is a possibility, but still results in 3 releases, so only worth it if
it is simpler

Thanks,
Danny

[1] https://lists.apache.org/thread/byy8fgkr8dbrrdv2nxpxkrz18h904r9b


On Mon, Sep 30, 2024 at 10:36 AM Arvid Heise <ar...@apache.org> wrote:

> Hi folks,
>
> thanks for the feedback. I'm still not convinced that it's the best
> approach to start with a bugfix release but I did a poor job in
> highlighting my thoughts.
> * I don't think we have any precedent of deprecating API in a bugfix
> release.
> * It's not obvious to me if a bugfix release counts towards our deprecation
> policy where we have one release between deprecation and removal. A user
> could use 3.2.0 not notice the 3.2.1 and see the deprecation only in 3.3.0.
> Then the user could be surprised by the removal in 4.0.0. (I'd personally
> see our policy too strict regarding major releases but that's what we did
> for Flink 2)
> * For the deprecation, I also introduced a new interface KafkaPartitioner
> that's a drop-in replacement for FlinkKafkaPartitioner. This is new API and
> should be part of a new minor release.
> * Finally, there is one new feature: [FLINK-22748] Allow dynamic target
> topic selection in SQL Kafka sinks [1].
> * I'd drop 1.18 still and support 1.20 instead.
> * I'd also bump kafka-clients to 3.7.1
>
> WDYT?
>
> Best,
>
> Arvid
>
> [1] https://github.com/apache/flink-connector-kafka/pull/109
>
> On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu <xb...@gmail.com> wrote:
>
> > Thanks Arvid for the volunteering!
> >
> > + 1 for all of the three releases and RM candidate.Qingsheng and I would
> > like to help the 4.0.0-preview which follows the Flink 2.0 preview,
> > please feel free to ping us if you need any help.
> >
> > Btw, for other external connectors which  highest supported flink version
> > still is 1.17 or 1.18,  I also hope we can have dedicated plan to bump
> > their version ASAP, we can
> > start a new thread to track other external connector releases.
> >
> > Best,
> > Leonard
> >
> >
> > > 2024年9月27日 下午6:54，Qingsheng Ren <re...@apache.org> 写道：
> > >
> > > Thanks for the work, Arvid!
> > >
> > > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If
> > > they are fully compatible, what about not dropping support for 1.18 in
> > > 3.2.1, and we release one more version 3.2.1-1.20? Then we can use
> > > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18
> > > and 1.19.
> > >
> > > And for the 4.0.0-preview version I'd like to help with it :-)
> > >
> > > Best,
> > > Qingsheng
> > >
> > > On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise <ar...@apache.org> wrote:
> > >>
> > >> Hi David,
> > >>
> > >> thank you very much for your reply.
> > >>
> > >>> Some thoughts on whether we need the 3 deliverables. And whether we
> > could
> > >> follow more traditional fixpack numbering:
> > >>> I see that there is already a release for 1.19
> > >>
> >
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> > >> .
> > >> I am wondering why we need the first deliverable. If we need it for
> the
> > bug
> > >> fixes , why not have a 3.2.1?
> > >> I forgot the most important part of the first release: drop Flink 1.18
> > >> support and add Flink 1.20 support. Hence I wouldn't want to mix that
> > into
> > >> a bugfix release. I think this is in line with the previous minor
> > releases.
> > >>
> > >>> I assume that kafka-3.4.0 will not work with previous Flink releases.
> > >> Would it be worth have a config switch to enable the lineage in the
> > >> connector so that we could use it with 1.19?  We could maybe do a 3.3
> if
> > >> this was the case.
> > >> Yes, as outlined in the discussion linked in the original message, we
> > need
> > >> to mixin new interfaces. Afaik classloading will fail if the
> interfaces
> > are
> > >> not present, even if the methods are not used. So I don't see how we
> can
> > >> use a config switch to make it happen (except with code duplication).
> > >> However, I'm grateful for any ideas to avoid this release.
> > >>
> > >> Best,
> > >>
> > >> Arvid
> > >>
> > >> On Fri, Sep 27, 2024 at 11:11 AM David Radley <
> david_radley@uk.ibm.com>
> > >> wrote:
> > >>
> > >>> Hi Arvid,
> > >>> Some thoughts on whether we need the 3 deliverables. And whether we
> > could
> > >>> follow more traditional fixpack numbering:
> > >>> I see that there is already a release for 1.19
> > >>>
> >
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> > >>> . I am wondering why we need the first deliverable. If we need it for
> > the
> > >>> bug fixes , why not have a 3.2.1?
> > >>> I assume that kafka-3.4.0 will not work with previous Flink releases.
> > >>> Would it be worth have a config switch to enable the lineage in the
> > >>> connector so that we could use it with 1.19?  We could maybe do a 3.3
> > if
> > >>> this was the case.
> > >>>
> > >>> WDYT?
> > >>>  Kind regards, David.
> > >>>
> > >>>
> > >>>
> > >>> From: Arvid Heise <ar...@apache.org>
> > >>> Date: Friday, 27 September 2024 at 09:24
> > >>> To: dev@flink.apache.org <de...@flink.apache.org>
> > >>> Subject: [EXTERNAL] Kafka connector releases
> > >>> Dear Flink devs,
> > >>>
> > >>> I'd like to initiate three(!) Kafka connector releases. The main
> > reason for
> > >>> having three releases is that we have been slacking a bit in keeping
> up
> > >>> with the latest changes.
> > >>>
> > >>> Here is the summary:
> > >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> > >>> - Incorporates lots of deprecations for Flink 2 including everything
> > that
> > >>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> > >>> (SourceFunction), and KafkaShuffle
> > >>> - Lots of bugfixes that are very relevant for 1.19 users (and
> probably
> > work
> > >>> with older releases)
> > >>>
> > >>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> > >>> - Incorporates lineage tracing which is only available in 1.20 [1]
> > (FLIP
> > >>> incorrectly says that it's avail in 1.19)
> > >>> - We have discussed some alternatives to this release in [2] but
> > basically
> > >>> having a separate release is the cleanest solution.
> > >>> - I'd like to linearize the releases to avoid having to do back or
> even
> > >>> forward ports
> > >>>
> > >>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> > >>> - Much requested to get the connector out asap for the preview. (I
> > think
> > >>> the old jar using the removed interfaces should still work)
> > >>> - Remove all deprecated things
> > >>> - General spring cleaning (trying to get rid of arch unit violations,
> > >>> migrate to JUnit5)
> > >>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> > >>>
> > >>> I'd appreciate any feedback and volunteers for RM ;) If you have
> > pending
> > >>> PRs that should be part of any of those releases, please also write
> > them.
> > >>>
> > >>> Best,
> > >>>
> > >>> Arvid
> > >>>
> > >>> [1]
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > >>> [1]
> > >>>
> > >>>
> >
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> > >>>
> > >>> Unless otherwise stated above:
> > >>>
> > >>> IBM United Kingdom Limited
> > >>> Registered in England and Wales with number 741598
> > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
> > >>>
> >
> >
>

Re: Kafka connector releases

[Arvid Heise <ar...@apache.org>]

Hi folks,

thanks for the feedback. I'm still not convinced that it's the best
approach to start with a bugfix release but I did a poor job in
highlighting my thoughts.
* I don't think we have any precedent of deprecating API in a bugfix
release.
* It's not obvious to me if a bugfix release counts towards our deprecation
policy where we have one release between deprecation and removal. A user
could use 3.2.0 not notice the 3.2.1 and see the deprecation only in 3.3.0.
Then the user could be surprised by the removal in 4.0.0. (I'd personally
see our policy too strict regarding major releases but that's what we did
for Flink 2)
* For the deprecation, I also introduced a new interface KafkaPartitioner
that's a drop-in replacement for FlinkKafkaPartitioner. This is new API and
should be part of a new minor release.
* Finally, there is one new feature: [FLINK-22748] Allow dynamic target
topic selection in SQL Kafka sinks [1].
* I'd drop 1.18 still and support 1.20 instead.
* I'd also bump kafka-clients to 3.7.1

WDYT?

Best,

Arvid

[1] https://github.com/apache/flink-connector-kafka/pull/109

On Sat, Sep 28, 2024 at 5:27 AM Leonard Xu <xb...@gmail.com> wrote:

> Thanks Arvid for the volunteering!
>
> + 1 for all of the three releases and RM candidate.Qingsheng and I would
> like to help the 4.0.0-preview which follows the Flink 2.0 preview,
> please feel free to ping us if you need any help.
>
> Btw, for other external connectors which  highest supported flink version
> still is 1.17 or 1.18,  I also hope we can have dedicated plan to bump
> their version ASAP, we can
> start a new thread to track other external connector releases.
>
> Best,
> Leonard
>
>
> > 2024年9月27日 下午6:54，Qingsheng Ren <re...@apache.org> 写道：
> >
> > Thanks for the work, Arvid!
> >
> > I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If
> > they are fully compatible, what about not dropping support for 1.18 in
> > 3.2.1, and we release one more version 3.2.1-1.20? Then we can use
> > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18
> > and 1.19.
> >
> > And for the 4.0.0-preview version I'd like to help with it :-)
> >
> > Best,
> > Qingsheng
> >
> > On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise <ar...@apache.org> wrote:
> >>
> >> Hi David,
> >>
> >> thank you very much for your reply.
> >>
> >>> Some thoughts on whether we need the 3 deliverables. And whether we
> could
> >> follow more traditional fixpack numbering:
> >>> I see that there is already a release for 1.19
> >>
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> >> .
> >> I am wondering why we need the first deliverable. If we need it for the
> bug
> >> fixes , why not have a 3.2.1?
> >> I forgot the most important part of the first release: drop Flink 1.18
> >> support and add Flink 1.20 support. Hence I wouldn't want to mix that
> into
> >> a bugfix release. I think this is in line with the previous minor
> releases.
> >>
> >>> I assume that kafka-3.4.0 will not work with previous Flink releases.
> >> Would it be worth have a config switch to enable the lineage in the
> >> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> >> this was the case.
> >> Yes, as outlined in the discussion linked in the original message, we
> need
> >> to mixin new interfaces. Afaik classloading will fail if the interfaces
> are
> >> not present, even if the methods are not used. So I don't see how we can
> >> use a config switch to make it happen (except with code duplication).
> >> However, I'm grateful for any ideas to avoid this release.
> >>
> >> Best,
> >>
> >> Arvid
> >>
> >> On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
> >> wrote:
> >>
> >>> Hi Arvid,
> >>> Some thoughts on whether we need the 3 deliverables. And whether we
> could
> >>> follow more traditional fixpack numbering:
> >>> I see that there is already a release for 1.19
> >>>
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> >>> . I am wondering why we need the first deliverable. If we need it for
> the
> >>> bug fixes , why not have a 3.2.1?
> >>> I assume that kafka-3.4.0 will not work with previous Flink releases.
> >>> Would it be worth have a config switch to enable the lineage in the
> >>> connector so that we could use it with 1.19?  We could maybe do a 3.3
> if
> >>> this was the case.
> >>>
> >>> WDYT?
> >>>  Kind regards, David.
> >>>
> >>>
> >>>
> >>> From: Arvid Heise <ar...@apache.org>
> >>> Date: Friday, 27 September 2024 at 09:24
> >>> To: dev@flink.apache.org <de...@flink.apache.org>
> >>> Subject: [EXTERNAL] Kafka connector releases
> >>> Dear Flink devs,
> >>>
> >>> I'd like to initiate three(!) Kafka connector releases. The main
> reason for
> >>> having three releases is that we have been slacking a bit in keeping up
> >>> with the latest changes.
> >>>
> >>> Here is the summary:
> >>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> >>> - Incorporates lots of deprecations for Flink 2 including everything
> that
> >>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> >>> (SourceFunction), and KafkaShuffle
> >>> - Lots of bugfixes that are very relevant for 1.19 users (and probably
> work
> >>> with older releases)
> >>>
> >>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> >>> - Incorporates lineage tracing which is only available in 1.20 [1]
> (FLIP
> >>> incorrectly says that it's avail in 1.19)
> >>> - We have discussed some alternatives to this release in [2] but
> basically
> >>> having a separate release is the cleanest solution.
> >>> - I'd like to linearize the releases to avoid having to do back or even
> >>> forward ports
> >>>
> >>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> >>> - Much requested to get the connector out asap for the preview. (I
> think
> >>> the old jar using the removed interfaces should still work)
> >>> - Remove all deprecated things
> >>> - General spring cleaning (trying to get rid of arch unit violations,
> >>> migrate to JUnit5)
> >>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
> >>>
> >>> I'd appreciate any feedback and volunteers for RM ;) If you have
> pending
> >>> PRs that should be part of any of those releases, please also write
> them.
> >>>
> >>> Best,
> >>>
> >>> Arvid
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> >>> [1]
> >>>
> >>>
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> >>>
> >>> Unless otherwise stated above:
> >>>
> >>> IBM United Kingdom Limited
> >>> Registered in England and Wales with number 741598
> >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >>>
>
>

Re: Kafka connector releases

[Leonard Xu <xb...@gmail.com>]

Thanks Arvid for the volunteering!

+ 1 for all of the three releases and RM candidate.Qingsheng and I would like to help the 4.0.0-preview which follows the Flink 2.0 preview,  
please feel free to ping us if you need any help.

Btw, for other external connectors which  highest supported flink version still is 1.17 or 1.18,  I also hope we can have dedicated plan to bump their version ASAP, we can 
start a new thread to track other external connector releases.

Best,
Leonard


> 2024年9月27日 下午6:54，Qingsheng Ren <re...@apache.org> 写道：
> 
> Thanks for the work, Arvid!
> 
> I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If
> they are fully compatible, what about not dropping support for 1.18 in
> 3.2.1, and we release one more version 3.2.1-1.20? Then we can use
> 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18
> and 1.19.
> 
> And for the 4.0.0-preview version I'd like to help with it :-)
> 
> Best,
> Qingsheng
> 
> On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise <ar...@apache.org> wrote:
>> 
>> Hi David,
>> 
>> thank you very much for your reply.
>> 
>>> Some thoughts on whether we need the 3 deliverables. And whether we could
>> follow more traditional fixpack numbering:
>>> I see that there is already a release for 1.19
>> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
>> .
>> I am wondering why we need the first deliverable. If we need it for the bug
>> fixes , why not have a 3.2.1?
>> I forgot the most important part of the first release: drop Flink 1.18
>> support and add Flink 1.20 support. Hence I wouldn't want to mix that into
>> a bugfix release. I think this is in line with the previous minor releases.
>> 
>>> I assume that kafka-3.4.0 will not work with previous Flink releases.
>> Would it be worth have a config switch to enable the lineage in the
>> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
>> this was the case.
>> Yes, as outlined in the discussion linked in the original message, we need
>> to mixin new interfaces. Afaik classloading will fail if the interfaces are
>> not present, even if the methods are not used. So I don't see how we can
>> use a config switch to make it happen (except with code duplication).
>> However, I'm grateful for any ideas to avoid this release.
>> 
>> Best,
>> 
>> Arvid
>> 
>> On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
>> wrote:
>> 
>>> Hi Arvid,
>>> Some thoughts on whether we need the 3 deliverables. And whether we could
>>> follow more traditional fixpack numbering:
>>> I see that there is already a release for 1.19
>>> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
>>> . I am wondering why we need the first deliverable. If we need it for the
>>> bug fixes , why not have a 3.2.1?
>>> I assume that kafka-3.4.0 will not work with previous Flink releases.
>>> Would it be worth have a config switch to enable the lineage in the
>>> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
>>> this was the case.
>>> 
>>> WDYT?
>>>  Kind regards, David.
>>> 
>>> 
>>> 
>>> From: Arvid Heise <ar...@apache.org>
>>> Date: Friday, 27 September 2024 at 09:24
>>> To: dev@flink.apache.org <de...@flink.apache.org>
>>> Subject: [EXTERNAL] Kafka connector releases
>>> Dear Flink devs,
>>> 
>>> I'd like to initiate three(!) Kafka connector releases. The main reason for
>>> having three releases is that we have been slacking a bit in keeping up
>>> with the latest changes.
>>> 
>>> Here is the summary:
>>> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
>>> - Incorporates lots of deprecations for Flink 2 including everything that
>>> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
>>> (SourceFunction), and KafkaShuffle
>>> - Lots of bugfixes that are very relevant for 1.19 users (and probably work
>>> with older releases)
>>> 
>>> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
>>> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
>>> incorrectly says that it's avail in 1.19)
>>> - We have discussed some alternatives to this release in [2] but basically
>>> having a separate release is the cleanest solution.
>>> - I'd like to linearize the releases to avoid having to do back or even
>>> forward ports
>>> 
>>> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
>>> - Much requested to get the connector out asap for the preview. (I think
>>> the old jar using the removed interfaces should still work)
>>> - Remove all deprecated things
>>> - General spring cleaning (trying to get rid of arch unit violations,
>>> migrate to JUnit5)
>>> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>>> 
>>> I'd appreciate any feedback and volunteers for RM ;) If you have pending
>>> PRs that should be part of any of those releases, please also write them.
>>> 
>>> Best,
>>> 
>>> Arvid
>>> 
>>> [1]
>>> 
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
>>> [1]
>>> 
>>> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>>> 
>>> Unless otherwise stated above:
>>> 
>>> IBM United Kingdom Limited
>>> Registered in England and Wales with number 741598
>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>>> 

Re: Kafka connector releases

[Qingsheng Ren <re...@apache.org>]

Thanks for the work, Arvid!

I'm not sure if there's any incompatibility issue for 3.2 + 1.20. If
they are fully compatible, what about not dropping support for 1.18 in
3.2.1, and we release one more version 3.2.1-1.20? Then we can use
3.3.0 for the new lineage feature in 1.20 and drop support for 1.18
and 1.19.

And for the 4.0.0-preview version I'd like to help with it :-)

Best,
Qingsheng

On Fri, Sep 27, 2024 at 6:13 PM Arvid Heise <ar...@apache.org> wrote:
>
> Hi David,
>
> thank you very much for your reply.
>
> > Some thoughts on whether we need the 3 deliverables. And whether we could
> follow more traditional fixpack numbering:
> > I see that there is already a release for 1.19
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> .
> I am wondering why we need the first deliverable. If we need it for the bug
> fixes , why not have a 3.2.1?
> I forgot the most important part of the first release: drop Flink 1.18
> support and add Flink 1.20 support. Hence I wouldn't want to mix that into
> a bugfix release. I think this is in line with the previous minor releases.
>
> > I assume that kafka-3.4.0 will not work with previous Flink releases.
> Would it be worth have a config switch to enable the lineage in the
> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> this was the case.
> Yes, as outlined in the discussion linked in the original message, we need
> to mixin new interfaces. Afaik classloading will fail if the interfaces are
> not present, even if the methods are not used. So I don't see how we can
> use a config switch to make it happen (except with code duplication).
> However, I'm grateful for any ideas to avoid this release.
>
> Best,
>
> Arvid
>
> On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
> wrote:
>
> > Hi Arvid,
> > Some thoughts on whether we need the 3 deliverables. And whether we could
> > follow more traditional fixpack numbering:
> > I see that there is already a release for 1.19
> > https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> > . I am wondering why we need the first deliverable. If we need it for the
> > bug fixes , why not have a 3.2.1?
> > I assume that kafka-3.4.0 will not work with previous Flink releases.
> > Would it be worth have a config switch to enable the lineage in the
> > connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> > this was the case.
> >
> > WDYT?
> >    Kind regards, David.
> >
> >
> >
> > From: Arvid Heise <ar...@apache.org>
> > Date: Friday, 27 September 2024 at 09:24
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [EXTERNAL] Kafka connector releases
> > Dear Flink devs,
> >
> > I'd like to initiate three(!) Kafka connector releases. The main reason for
> > having three releases is that we have been slacking a bit in keeping up
> > with the latest changes.
> >
> > Here is the summary:
> > 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> > - Incorporates lots of deprecations for Flink 2 including everything that
> > is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> > (SourceFunction), and KafkaShuffle
> > - Lots of bugfixes that are very relevant for 1.19 users (and probably work
> > with older releases)
> >
> > 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> > - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> > incorrectly says that it's avail in 1.19)
> > - We have discussed some alternatives to this release in [2] but basically
> > having a separate release is the cleanest solution.
> > - I'd like to linearize the releases to avoid having to do back or even
> > forward ports
> >
> > 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> > - Much requested to get the connector out asap for the preview. (I think
> > the old jar using the removed interfaces should still work)
> > - Remove all deprecated things
> > - General spring cleaning (trying to get rid of arch unit violations,
> > migrate to JUnit5)
> > - Should we relocate the TableAPI stuff to o.a.f.connectors?
> >
> > I'd appreciate any feedback and volunteers for RM ;) If you have pending
> > PRs that should be part of any of those releases, please also write them.
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > [1]
> >
> > https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >

FW: RE: Kafka connector releases

[David Radley <da...@uk.ibm.com>]

Hi again Arvid,
I realise we actually have 3 releases (2 bug releases) supporting 1.18 – the renumbering threw me 😊. So I think your proposal looks good,      kind regards, David.


From: David Radley <da...@uk.ibm.com>
Date: Monday, 30 September 2024 at 17:07
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] RE: Kafka connector releases
Hi Arvid,
Thanks for your reply on this.

So we could do a 3.2.1, if we supported Flink 1.18? Is there a compelling reason to not support 1.18?

I see that  Kafka connectors that supported Flink 1.17 and below had 2 to 6 bug releases. Is there not an expectation that there will be a Flink 1.18 compatible Kafka connector bug release? If we end up having to do a 1.18 bug release due to demand and expectation, then maybe a 3.2.1 supporting 1.18 would make sense.

It seems like we are changing the way we are doing the bug releases for the Kafka connector. Is the idea that there will be no bug release for 1.19 also? I assume 1.20 as the last v1 release would have bug releases.

Thoughts?

Kind regards, David.


From: Arvid Heise <ar...@apache.org>
Date: Friday, 27 September 2024 at 11:12
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: Kafka connector releases
Hi David,

thank you very much for your reply.

> Some thoughts on whether we need the 3 deliverables. And whether we could
follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
.
I am wondering why we need the first deliverable. If we need it for the bug
fixes , why not have a 3.2.1?
I forgot the most important part of the first release: drop Flink 1.18
support and add Flink 1.20 support. Hence I wouldn't want to mix that into
a bugfix release. I think this is in line with the previous minor releases.

> I assume that kafka-3.4.0 will not work with previous Flink releases.
Would it be worth have a config switch to enable the lineage in the
connector so that we could use it with 1.19?  We could maybe do a 3.3 if
this was the case.
Yes, as outlined in the discussion linked in the original message, we need
to mixin new interfaces. Afaik classloading will fail if the interfaces are
not present, even if the methods are not used. So I don't see how we can
use a config switch to make it happen (except with code duplication).
However, I'm grateful for any ideas to avoid this release.

Best,

Arvid

On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Arvid,
> Some thoughts on whether we need the 3 deliverables. And whether we could
> follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> . I am wondering why we need the first deliverable. If we need it for the
> bug fixes , why not have a 3.2.1?
> I assume that kafka-3.4.0 will not work with previous Flink releases.
> Would it be worth have a config switch to enable the lineage in the
> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> this was the case.
>
> WDYT?
>    Kind regards, David.
>
>
>
> From: Arvid Heise <ar...@apache.org>
> Date: Friday, 27 September 2024 at 09:24
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Kafka connector releases
> Dear Flink devs,
>
> I'd like to initiate three(!) Kafka connector releases. The main reason for
> having three releases is that we have been slacking a bit in keeping up
> with the latest changes.
>
> Here is the summary:
> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> - Incorporates lots of deprecations for Flink 2 including everything that
> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> (SourceFunction), and KafkaShuffle
> - Lots of bugfixes that are very relevant for 1.19 users (and probably work
> with older releases)
>
> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> incorrectly says that it's avail in 1.19)
> - We have discussed some alternatives to this release in [2] but basically
> having a separate release is the cleanest solution.
> - I'd like to linearize the releases to avoid having to do back or even
> forward ports
>
> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> - Much requested to get the connector out asap for the preview. (I think
> the old jar using the removed interfaces should still work)
> - Remove all deprecated things
> - General spring cleaning (trying to get rid of arch unit violations,
> migrate to JUnit5)
> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>
> I'd appreciate any feedback and volunteers for RM ;) If you have pending
> PRs that should be part of any of those releases, please also write them.
>
> Best,
>
> Arvid
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> [1]
>
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

RE: Kafka connector releases

[David Radley <da...@uk.ibm.com>]

Hi Arvid,
Thanks for your reply on this.

So we could do a 3.2.1, if we supported Flink 1.18? Is there a compelling reason to not support 1.18?

I see that  Kafka connectors that supported Flink 1.17 and below had 2 to 6 bug releases. Is there not an expectation that there will be a Flink 1.18 compatible Kafka connector bug release? If we end up having to do a 1.18 bug release due to demand and expectation, then maybe a 3.2.1 supporting 1.18 would make sense.

It seems like we are changing the way we are doing the bug releases for the Kafka connector. Is the idea that there will be no bug release for 1.19 also? I assume 1.20 as the last v1 release would have bug releases.

Thoughts?

Kind regards, David.


From: Arvid Heise <ar...@apache.org>
Date: Friday, 27 September 2024 at 11:12
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: Kafka connector releases
Hi David,

thank you very much for your reply.

> Some thoughts on whether we need the 3 deliverables. And whether we could
follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
.
I am wondering why we need the first deliverable. If we need it for the bug
fixes , why not have a 3.2.1?
I forgot the most important part of the first release: drop Flink 1.18
support and add Flink 1.20 support. Hence I wouldn't want to mix that into
a bugfix release. I think this is in line with the previous minor releases.

> I assume that kafka-3.4.0 will not work with previous Flink releases.
Would it be worth have a config switch to enable the lineage in the
connector so that we could use it with 1.19?  We could maybe do a 3.3 if
this was the case.
Yes, as outlined in the discussion linked in the original message, we need
to mixin new interfaces. Afaik classloading will fail if the interfaces are
not present, even if the methods are not used. So I don't see how we can
use a config switch to make it happen (except with code duplication).
However, I'm grateful for any ideas to avoid this release.

Best,

Arvid

On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Arvid,
> Some thoughts on whether we need the 3 deliverables. And whether we could
> follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> . I am wondering why we need the first deliverable. If we need it for the
> bug fixes , why not have a 3.2.1?
> I assume that kafka-3.4.0 will not work with previous Flink releases.
> Would it be worth have a config switch to enable the lineage in the
> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> this was the case.
>
> WDYT?
>    Kind regards, David.
>
>
>
> From: Arvid Heise <ar...@apache.org>
> Date: Friday, 27 September 2024 at 09:24
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Kafka connector releases
> Dear Flink devs,
>
> I'd like to initiate three(!) Kafka connector releases. The main reason for
> having three releases is that we have been slacking a bit in keeping up
> with the latest changes.
>
> Here is the summary:
> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> - Incorporates lots of deprecations for Flink 2 including everything that
> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> (SourceFunction), and KafkaShuffle
> - Lots of bugfixes that are very relevant for 1.19 users (and probably work
> with older releases)
>
> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> incorrectly says that it's avail in 1.19)
> - We have discussed some alternatives to this release in [2] but basically
> having a separate release is the cleanest solution.
> - I'd like to linearize the releases to avoid having to do back or even
> forward ports
>
> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> - Much requested to get the connector out asap for the preview. (I think
> the old jar using the removed interfaces should still work)
> - Remove all deprecated things
> - General spring cleaning (trying to get rid of arch unit violations,
> migrate to JUnit5)
> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>
> I'd appreciate any feedback and volunteers for RM ;) If you have pending
> PRs that should be part of any of those releases, please also write them.
>
> Best,
>
> Arvid
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> [1]
>
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: Kafka connector releases

[Arvid Heise <ar...@apache.org>]

Hi David,

thank you very much for your reply.

> Some thoughts on whether we need the 3 deliverables. And whether we could
follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
.
I am wondering why we need the first deliverable. If we need it for the bug
fixes , why not have a 3.2.1?
I forgot the most important part of the first release: drop Flink 1.18
support and add Flink 1.20 support. Hence I wouldn't want to mix that into
a bugfix release. I think this is in line with the previous minor releases.

> I assume that kafka-3.4.0 will not work with previous Flink releases.
Would it be worth have a config switch to enable the lineage in the
connector so that we could use it with 1.19?  We could maybe do a 3.3 if
this was the case.
Yes, as outlined in the discussion linked in the original message, we need
to mixin new interfaces. Afaik classloading will fail if the interfaces are
not present, even if the methods are not used. So I don't see how we can
use a config switch to make it happen (except with code duplication).
However, I'm grateful for any ideas to avoid this release.

Best,

Arvid

On Fri, Sep 27, 2024 at 11:11 AM David Radley <da...@uk.ibm.com>
wrote:

> Hi Arvid,
> Some thoughts on whether we need the 3 deliverables. And whether we could
> follow more traditional fixpack numbering:
> I see that there is already a release for 1.19
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19
> . I am wondering why we need the first deliverable. If we need it for the
> bug fixes , why not have a 3.2.1?
> I assume that kafka-3.4.0 will not work with previous Flink releases.
> Would it be worth have a config switch to enable the lineage in the
> connector so that we could use it with 1.19?  We could maybe do a 3.3 if
> this was the case.
>
> WDYT?
>    Kind regards, David.
>
>
>
> From: Arvid Heise <ar...@apache.org>
> Date: Friday, 27 September 2024 at 09:24
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Kafka connector releases
> Dear Flink devs,
>
> I'd like to initiate three(!) Kafka connector releases. The main reason for
> having three releases is that we have been slacking a bit in keeping up
> with the latest changes.
>
> Here is the summary:
> 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
> - Incorporates lots of deprecations for Flink 2 including everything that
> is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
> (SourceFunction), and KafkaShuffle
> - Lots of bugfixes that are very relevant for 1.19 users (and probably work
> with older releases)
>
> 2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
> - Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
> incorrectly says that it's avail in 1.19)
> - We have discussed some alternatives to this release in [2] but basically
> having a separate release is the cleanest solution.
> - I'd like to linearize the releases to avoid having to do back or even
> forward ports
>
> 3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
> - Much requested to get the connector out asap for the preview. (I think
> the old jar using the removed interfaces should still work)
> - Remove all deprecated things
> - General spring cleaning (trying to get rid of arch unit violations,
> migrate to JUnit5)
> - Should we relocate the TableAPI stuff to o.a.f.connectors?
>
> I'd appreciate any feedback and volunteers for RM ;) If you have pending
> PRs that should be part of any of those releases, please also write them.
>
> Best,
>
> Arvid
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> [1]
>
> https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: Kafka connector releases

[David Radley <da...@uk.ibm.com>]

Hi Arvid,
Some thoughts on whether we need the 3 deliverables. And whether we could follow more traditional fixpack numbering:
I see that there is already a release for 1.19 https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.2.0-1.19 . I am wondering why we need the first deliverable. If we need it for the bug fixes , why not have a 3.2.1?
I assume that kafka-3.4.0 will not work with previous Flink releases. Would it be worth have a config switch to enable the lineage in the connector so that we could use it with 1.19?  We could maybe do a 3.3 if this was the case.

WDYT?
   Kind regards, David.



From: Arvid Heise <ar...@apache.org>
Date: Friday, 27 September 2024 at 09:24
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Kafka connector releases
Dear Flink devs,

I'd like to initiate three(!) Kafka connector releases. The main reason for
having three releases is that we have been slacking a bit in keeping up
with the latest changes.

Here is the summary:
1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap)
- Incorporates lots of deprecations for Flink 2 including everything that
is related to FlinkKafkaProducer (SinkFunction, FlinkKafkaConsumer
(SourceFunction), and KafkaShuffle
- Lots of bugfixes that are very relevant for 1.19 users (and probably work
with older releases)

2. Release kafka-3.4.0 targeting 1.20 (~1-2 weeks later)
- Incorporates lineage tracing which is only available in 1.20 [1] (FLIP
incorrectly says that it's avail in 1.19)
- We have discussed some alternatives to this release in [2] but basically
having a separate release is the cleanest solution.
- I'd like to linearize the releases to avoid having to do back or even
forward ports

3. Release kafka-4.0.0 targeting 2.0-preview (~1-2 weeks later)
- Much requested to get the connector out asap for the preview. (I think
the old jar using the removed interfaces should still work)
- Remove all deprecated things
- General spring cleaning (trying to get rid of arch unit violations,
migrate to JUnit5)
- Should we relocate the TableAPI stuff to o.a.f.connectors?

I'd appreciate any feedback and volunteers for RM ;) If you have pending
PRs that should be part of any of those releases, please also write them.

Best,

Arvid

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
[1]
https://github.com/apache/flink-connector-kafka/pull/111#issuecomment-2306382878

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU