You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Martijn Visser <ma...@ververica.com> on 2022/02/18 13:54:21 UTC

[DISCUSS] Plan to externalize connectors and versioning

Hi everyone,

As a follow-up to earlier discussions [1] [2] to externalize the connectors
from the Flink repository, I would like to propose a plan to externalize
these connectors. The goal of this plan is to start with moving connectors
to its own repositories without introducing regressions for connector
developers.

The plan is as follows:

1. A new repository is requested for a connector.
2. The code for that connector is moved to its individual repository,
including the commit history
3. Any first release made for a connector in an external connector
repository starts with major version 3, so 3.0.0. The reason for that is
that we want to decouple the Flink releases from a connector release. If we
would start with major version 2, it could cause some confusion because
people could think a Flink 2.0 has been released. This does mean that each
connector needs to have a compatibility matrix generated, stating which
version number of the connector is compatible with the correct Flink
version.
4. The group id and artifact id for the connector will remain the same,
only the version is different.
5. The connector dependencies on the Flink website are updated to point to
the newly released connector artifact.
6. If a connector is moved, there is one release cycle where there will be
binary releases for that connector in both Flink core and from the
connector repository. This is to make Flink users who are upgrading
slightly easier. We will have to make a note in the release notes that a
connector has been moved and that a user should update any references from
the original connector artifact (from the Flink connector) to the new
connector artifact (from the external conenctor version)

We propose to first try to execute this plan for the Elasticsearch
connector as follows:

1. We wait until the Flink 1.15 release branch is cut
2. When that's done, the Elasticsearch code (including commit history) from
Flink's 1.15 release branch will be moved to the
flink-connector-elasticsearch main branch.
3. When Flink 1.15 is released, we will also release an Elasticsearch
connector for the external connector repository with version 3.0.0.
4. Bugfixes or improvements will be made first pointing to the external
connector repository and will be cherry-picked back to the release-1.15
branch in the Flink core repository.
5. The Elasticsearch code, test etc will be removed from the master branch
in the Flink core repository and dropped with Flink 1.16

Looking forward to your thoughts on this!

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
[2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Martijn Visser <ma...@ververica.com>.

Thanks for clarifying my story Konstantin, this was indeed as we discussed.
Thanks Chesnay and Thomas for your input too!

On Thu, 24 Feb 2022 at 02:43, Thomas Weise <th...@apache.org> wrote:

> This plan LGTM.
>
> Thanks,
> Thomas
>
> On Wed, Feb 23, 2022 at 4:28 AM Chesnay Schepler <ch...@apache.org>
> wrote:
> >
> > That sounds fine to me.
> >
> > On 23/02/2022 10:49, Konstantin Knauf wrote:
> > > Hi Chesnay, Hi everyone,
> > >
> > > I think the idea for the migration is the following (with the example
> of
> > > ElasticSearch). I talked to Martijn offline.
> > >
> > > 1. ElasticSearch Connector is released from the core repository with
> the
> > > Flink 1.15.0 release. No changes.
> > >
> > > 2. At the beginning of the Flink 1.16 release cycle the connector is
> > > removed from `master` of the core repository. It remains on the
> > > `release-1.15` branch and earlier release branches.
> > >
> > > 3. The connector code is "copied" over to the `master` branch of a
> > > `flink-connector-elastic-search` repository. Bugfixes to the connector
> need
> > > to go to both `release-1.15` and before in the core repository and
> `master`
> > > of the external repository.
> > >
> > > 4. Once all the processes required to do a release in the
> > > `flink-connector-elastic-search` are in place (docs integration,
> release
> > > automation,...), we release flink-connector-elastic-search:3.0.0, which
> > > will be compatible with Flink 1.15. At this point, users can choose
> whether
> > > they use flink-connector-elastic-search:1.15.x (released from the core
> > > repository) or flink-connector-elastic-search:3.0.0 already released
> from
> > > the external repository with Flink 1.15. The documentation will already
> > > advertise the one released from the external repository. This is the
> > > "overlap" that Martijn mentioned.
> > >
> > > 5. From here onwards, the release cycle of the ElasticSearch Connector
> is
> > > independent. There could be 3.1.0 and 3.0.1 etc. The compatibility
> matrix
> > > will be part of the connector documentation.
> > >
> > > 6. If there is a patch release for Flink 1.15-, this will of course
> also
> > > include flink-connector-elastic-search release from the core
> repository.
> > >
> > > 7. For Flink 1.16, there might or might not be a release of the
> > > elastic-search-connector from the external repository. Depends on
> > > compatibility.
> > >
> > > I hope this clarifies it a bit and it makes sense to me.
> > >
> > > It also makes sense to me to do this as soon as possible (probably
> once the
> > > release-1.15 branch is cut) with the example of ElasticSearch.
> Afterwards
> > > (hopefully still in the Flink 1.16 release cycle) we do the same for
> other
> > > connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or
> > > helpful to make it a condition that this happens for all connectors at
> the
> > > same time.
> > >
> > > Cheers and thanks,
> > >
> > > Konstantin
> > >
> > >
> > > On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ch...@apache.org>
> wrote:
> > >
> > >> If we don't make a release, I think it would appear as partially
> > >> externalized (since the binaries are still only created with Flink
> core,
> > >> not from the external repository).
> > >>
> > >> I'm wondering you are referring to when you say "it appear[s]". Users
> > >> don't know about it, and contributors can be easily informed about the
> > >> current state. Who's opinion are you worried about?
> > >>
> > >> doing a release [means] that we have then completely externalized the
> > >> connector
> > >>
> > >> In my mind the connector is completely externalized once the connector
> > >> project is complete and can act independently from the core repo. That
> > >> includes having all the code, working CI and the documentation being
> > >> integrated into the Flink website. And /then/ we can do a release. I
> > >> don't see how this could work any other way; how could we possibly
> argue
> > >> that the connector is externalized when development on the connector
> > >> isn't even possible in that repository?
> > >>
> > >> There are also other connectors (like Opensearch and I believe
> RabbitMQ)
> > >> that will end up straight in their own repositories
> > >>
> > >>
> > >> Which is a bit of a different situation because here the only source
> of
> > >> this connector will have been that repository.
> > >>
> > >> Would you prefer to remove a connector in a Flink patch release?
> > >>
> > >>
> > >> No. I think I misread your statement; when you said that there "is 1
> > >> release cycle where the connector both exists in Flink core and the
> > >> external repo", you are referring to 1.15, correct? (although this
> > >> should also apply to 1.14 so isn't it 2 releases...?)
> > >> How I understood it was that we'd keep the connector around until
> 1.16,
> > >> which would obviously be terrible.
> > >>
> > >> On 19/02/2022 13:30, Martijn Visser wrote:
> > >>> Hi Chesnay,
> > >>>
> > >>> I think the advantage of also doing a release is that we have then
> > >>> completely externalized the connector. If we don't make a release, I
> > >> think
> > >>> it would appear as partially externalized (since the binaries are
> still
> > >>> only created with Flink core, not from the external repository). It
> would
> > >>> also mean that in our documentation we would still point to the
> binary
> > >>> created with the Flink core release.
> > >>>
> > >>> There are also other connectors (like Opensearch and I believe
> RabbitMQ)
> > >>> that will end up straight in their own repositories. Since we also
> would
> > >>> like to document those, I don't think the situation will be messy.
> We can
> > >>> also solve it with an information hint in the documentation.
> > >>>
> > >>> With regards to point 6, do you have an alternative? Would you
> prefer to
> > >>> remove a connector in a Flink patch release?
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Martijn
> > >>>
> > >>> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ch...@apache.org>
> > >> wrote:
> > >>>> Why do you want to immediately do a release for the elasticsearch
> > >>>> connector? What does that provide us?
> > >>>>
> > >>>> I'd rather first have a fully working setup and integrated
> documentation
> > >>>> before thinking about releasing anything.
> > >>>> Once we have that we may be able to externalize all connectors
> within 1
> > >>>> release cycle and do a clean switch; otherwise we end up with a bit
> of a
> > >>>> messy situation for users where some connectors use version scheme
> A and
> > >>>> others B.
> > >>>>
> > >>>> I also doubt the value of 6). They'll have to update the version
> anyway
> > >>>> (and discover at some point that the version scheme has changed),
> so I
> > >>>> don't see what this makes easier.
> > >>>>
> > >>>> On 18/02/2022 14:54, Martijn Visser wrote:
> > >>>>> Hi everyone,
> > >>>>>
> > >>>>> As a follow-up to earlier discussions [1] [2] to externalize the
> > >>>> connectors
> > >>>>> from the Flink repository, I would like to propose a plan to
> > >> externalize
> > >>>>> these connectors. The goal of this plan is to start with moving
> > >>>> connectors
> > >>>>> to its own repositories without introducing regressions for
> connector
> > >>>>> developers.
> > >>>>>
> > >>>>> The plan is as follows:
> > >>>>>
> > >>>>> 1. A new repository is requested for a connector.
> > >>>>> 2. The code for that connector is moved to its individual
> repository,
> > >>>>> including the commit history
> > >>>>> 3. Any first release made for a connector in an external connector
> > >>>>> repository starts with major version 3, so 3.0.0. The reason for
> that
> > >> is
> > >>>>> that we want to decouple the Flink releases from a connector
> release.
> > >> If
> > >>>> we
> > >>>>> would start with major version 2, it could cause some confusion
> because
> > >>>>> people could think a Flink 2.0 has been released. This does mean
> that
> > >>>> each
> > >>>>> connector needs to have a compatibility matrix generated, stating
> which
> > >>>>> version number of the connector is compatible with the correct
> Flink
> > >>>>> version.
> > >>>>> 4. The group id and artifact id for the connector will remain the
> same,
> > >>>>> only the version is different.
> > >>>>> 5. The connector dependencies on the Flink website are updated to
> point
> > >>>> to
> > >>>>> the newly released connector artifact.
> > >>>>> 6. If a connector is moved, there is one release cycle where there
> will
> > >>>> be
> > >>>>> binary releases for that connector in both Flink core and from the
> > >>>>> connector repository. This is to make Flink users who are upgrading
> > >>>>> slightly easier. We will have to make a note in the release notes
> that
> > >> a
> > >>>>> connector has been moved and that a user should update any
> references
> > >>>> from
> > >>>>> the original connector artifact (from the Flink connector) to the
> new
> > >>>>> connector artifact (from the external conenctor version)
> > >>>>>
> > >>>>> We propose to first try to execute this plan for the Elasticsearch
> > >>>>> connector as follows:
> > >>>>>
> > >>>>> 1. We wait until the Flink 1.15 release branch is cut
> > >>>>> 2. When that's done, the Elasticsearch code (including commit
> history)
> > >>>> from
> > >>>>> Flink's 1.15 release branch will be moved to the
> > >>>>> flink-connector-elasticsearch main branch.
> > >>>>> 3. When Flink 1.15 is released, we will also release an
> Elasticsearch
> > >>>>> connector for the external connector repository with version 3.0.0.
> > >>>>> 4. Bugfixes or improvements will be made first pointing to the
> external
> > >>>>> connector repository and will be cherry-picked back to the
> release-1.15
> > >>>>> branch in the Flink core repository.
> > >>>>> 5. The Elasticsearch code, test etc will be removed from the master
> > >>>> branch
> > >>>>> in the Flink core repository and dropped with Flink 1.16
> > >>>>>
> > >>>>> Looking forward to your thoughts on this!
> > >>>>>
> > >>>>> Best regards,
> > >>>>>
> > >>>>> Martijn Visser
> > >>>>> https://twitter.com/MartijnVisser82
> > >>>>>
> > >>>>> [1]
> https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> > >>>>> [2]
> https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
> > >>>>>
> > >
> >
>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Thomas Weise <th...@apache.org>.

This plan LGTM.

Thanks,
Thomas

On Wed, Feb 23, 2022 at 4:28 AM Chesnay Schepler <ch...@apache.org> wrote:
>
> That sounds fine to me.
>
> On 23/02/2022 10:49, Konstantin Knauf wrote:
> > Hi Chesnay, Hi everyone,
> >
> > I think the idea for the migration is the following (with the example of
> > ElasticSearch). I talked to Martijn offline.
> >
> > 1. ElasticSearch Connector is released from the core repository with the
> > Flink 1.15.0 release. No changes.
> >
> > 2. At the beginning of the Flink 1.16 release cycle the connector is
> > removed from `master` of the core repository. It remains on the
> > `release-1.15` branch and earlier release branches.
> >
> > 3. The connector code is "copied" over to the `master` branch of a
> > `flink-connector-elastic-search` repository. Bugfixes to the connector need
> > to go to both `release-1.15` and before in the core repository and `master`
> > of the external repository.
> >
> > 4. Once all the processes required to do a release in the
> > `flink-connector-elastic-search` are in place (docs integration, release
> > automation,...), we release flink-connector-elastic-search:3.0.0, which
> > will be compatible with Flink 1.15. At this point, users can choose whether
> > they use flink-connector-elastic-search:1.15.x (released from the core
> > repository) or flink-connector-elastic-search:3.0.0 already released from
> > the external repository with Flink 1.15. The documentation will already
> > advertise the one released from the external repository. This is the
> > "overlap" that Martijn mentioned.
> >
> > 5. From here onwards, the release cycle of the ElasticSearch Connector is
> > independent. There could be 3.1.0 and 3.0.1 etc. The compatibility matrix
> > will be part of the connector documentation.
> >
> > 6. If there is a patch release for Flink 1.15-, this will of course also
> > include flink-connector-elastic-search release from the core repository.
> >
> > 7. For Flink 1.16, there might or might not be a release of the
> > elastic-search-connector from the external repository. Depends on
> > compatibility.
> >
> > I hope this clarifies it a bit and it makes sense to me.
> >
> > It also makes sense to me to do this as soon as possible (probably once the
> > release-1.15 branch is cut) with the example of ElasticSearch. Afterwards
> > (hopefully still in the Flink 1.16 release cycle) we do the same for other
> > connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or
> > helpful to make it a condition that this happens for all connectors at the
> > same time.
> >
> > Cheers and thanks,
> >
> > Konstantin
> >
> >
> > On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ch...@apache.org> wrote:
> >
> >> If we don't make a release, I think it would appear as partially
> >> externalized (since the binaries are still only created with Flink core,
> >> not from the external repository).
> >>
> >> I'm wondering you are referring to when you say "it appear[s]". Users
> >> don't know about it, and contributors can be easily informed about the
> >> current state. Who's opinion are you worried about?
> >>
> >> doing a release [means] that we have then completely externalized the
> >> connector
> >>
> >> In my mind the connector is completely externalized once the connector
> >> project is complete and can act independently from the core repo. That
> >> includes having all the code, working CI and the documentation being
> >> integrated into the Flink website. And /then/ we can do a release. I
> >> don't see how this could work any other way; how could we possibly argue
> >> that the connector is externalized when development on the connector
> >> isn't even possible in that repository?
> >>
> >> There are also other connectors (like Opensearch and I believe RabbitMQ)
> >> that will end up straight in their own repositories
> >>
> >>
> >> Which is a bit of a different situation because here the only source of
> >> this connector will have been that repository.
> >>
> >> Would you prefer to remove a connector in a Flink patch release?
> >>
> >>
> >> No. I think I misread your statement; when you said that there "is 1
> >> release cycle where the connector both exists in Flink core and the
> >> external repo", you are referring to 1.15, correct? (although this
> >> should also apply to 1.14 so isn't it 2 releases...?)
> >> How I understood it was that we'd keep the connector around until 1.16,
> >> which would obviously be terrible.
> >>
> >> On 19/02/2022 13:30, Martijn Visser wrote:
> >>> Hi Chesnay,
> >>>
> >>> I think the advantage of also doing a release is that we have then
> >>> completely externalized the connector. If we don't make a release, I
> >> think
> >>> it would appear as partially externalized (since the binaries are still
> >>> only created with Flink core, not from the external repository). It would
> >>> also mean that in our documentation we would still point to the binary
> >>> created with the Flink core release.
> >>>
> >>> There are also other connectors (like Opensearch and I believe RabbitMQ)
> >>> that will end up straight in their own repositories. Since we also would
> >>> like to document those, I don't think the situation will be messy. We can
> >>> also solve it with an information hint in the documentation.
> >>>
> >>> With regards to point 6, do you have an alternative? Would you prefer to
> >>> remove a connector in a Flink patch release?
> >>>
> >>> Best regards,
> >>>
> >>> Martijn
> >>>
> >>> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ch...@apache.org>
> >> wrote:
> >>>> Why do you want to immediately do a release for the elasticsearch
> >>>> connector? What does that provide us?
> >>>>
> >>>> I'd rather first have a fully working setup and integrated documentation
> >>>> before thinking about releasing anything.
> >>>> Once we have that we may be able to externalize all connectors within 1
> >>>> release cycle and do a clean switch; otherwise we end up with a bit of a
> >>>> messy situation for users where some connectors use version scheme A and
> >>>> others B.
> >>>>
> >>>> I also doubt the value of 6). They'll have to update the version anyway
> >>>> (and discover at some point that the version scheme has changed), so I
> >>>> don't see what this makes easier.
> >>>>
> >>>> On 18/02/2022 14:54, Martijn Visser wrote:
> >>>>> Hi everyone,
> >>>>>
> >>>>> As a follow-up to earlier discussions [1] [2] to externalize the
> >>>> connectors
> >>>>> from the Flink repository, I would like to propose a plan to
> >> externalize
> >>>>> these connectors. The goal of this plan is to start with moving
> >>>> connectors
> >>>>> to its own repositories without introducing regressions for connector
> >>>>> developers.
> >>>>>
> >>>>> The plan is as follows:
> >>>>>
> >>>>> 1. A new repository is requested for a connector.
> >>>>> 2. The code for that connector is moved to its individual repository,
> >>>>> including the commit history
> >>>>> 3. Any first release made for a connector in an external connector
> >>>>> repository starts with major version 3, so 3.0.0. The reason for that
> >> is
> >>>>> that we want to decouple the Flink releases from a connector release.
> >> If
> >>>> we
> >>>>> would start with major version 2, it could cause some confusion because
> >>>>> people could think a Flink 2.0 has been released. This does mean that
> >>>> each
> >>>>> connector needs to have a compatibility matrix generated, stating which
> >>>>> version number of the connector is compatible with the correct Flink
> >>>>> version.
> >>>>> 4. The group id and artifact id for the connector will remain the same,
> >>>>> only the version is different.
> >>>>> 5. The connector dependencies on the Flink website are updated to point
> >>>> to
> >>>>> the newly released connector artifact.
> >>>>> 6. If a connector is moved, there is one release cycle where there will
> >>>> be
> >>>>> binary releases for that connector in both Flink core and from the
> >>>>> connector repository. This is to make Flink users who are upgrading
> >>>>> slightly easier. We will have to make a note in the release notes that
> >> a
> >>>>> connector has been moved and that a user should update any references
> >>>> from
> >>>>> the original connector artifact (from the Flink connector) to the new
> >>>>> connector artifact (from the external conenctor version)
> >>>>>
> >>>>> We propose to first try to execute this plan for the Elasticsearch
> >>>>> connector as follows:
> >>>>>
> >>>>> 1. We wait until the Flink 1.15 release branch is cut
> >>>>> 2. When that's done, the Elasticsearch code (including commit history)
> >>>> from
> >>>>> Flink's 1.15 release branch will be moved to the
> >>>>> flink-connector-elasticsearch main branch.
> >>>>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
> >>>>> connector for the external connector repository with version 3.0.0.
> >>>>> 4. Bugfixes or improvements will be made first pointing to the external
> >>>>> connector repository and will be cherry-picked back to the release-1.15
> >>>>> branch in the Flink core repository.
> >>>>> 5. The Elasticsearch code, test etc will be removed from the master
> >>>> branch
> >>>>> in the Flink core repository and dropped with Flink 1.16
> >>>>>
> >>>>> Looking forward to your thoughts on this!
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> Martijn Visser
> >>>>> https://twitter.com/MartijnVisser82
> >>>>>
> >>>>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> >>>>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
> >>>>>
> >
>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Chesnay Schepler <ch...@apache.org>.

That sounds fine to me.

On 23/02/2022 10:49, Konstantin Knauf wrote:
> Hi Chesnay, Hi everyone,
>
> I think the idea for the migration is the following (with the example of
> ElasticSearch). I talked to Martijn offline.
>
> 1. ElasticSearch Connector is released from the core repository with the
> Flink 1.15.0 release. No changes.
>
> 2. At the beginning of the Flink 1.16 release cycle the connector is
> removed from `master` of the core repository. It remains on the
> `release-1.15` branch and earlier release branches.
>
> 3. The connector code is "copied" over to the `master` branch of a
> `flink-connector-elastic-search` repository. Bugfixes to the connector need
> to go to both `release-1.15` and before in the core repository and `master`
> of the external repository.
>
> 4. Once all the processes required to do a release in the
> `flink-connector-elastic-search` are in place (docs integration, release
> automation,...), we release flink-connector-elastic-search:3.0.0, which
> will be compatible with Flink 1.15. At this point, users can choose whether
> they use flink-connector-elastic-search:1.15.x (released from the core
> repository) or flink-connector-elastic-search:3.0.0 already released from
> the external repository with Flink 1.15. The documentation will already
> advertise the one released from the external repository. This is the
> "overlap" that Martijn mentioned.
>
> 5. From here onwards, the release cycle of the ElasticSearch Connector is
> independent. There could be 3.1.0 and 3.0.1 etc. The compatibility matrix
> will be part of the connector documentation.
>
> 6. If there is a patch release for Flink 1.15-, this will of course also
> include flink-connector-elastic-search release from the core repository.
>
> 7. For Flink 1.16, there might or might not be a release of the
> elastic-search-connector from the external repository. Depends on
> compatibility.
>
> I hope this clarifies it a bit and it makes sense to me.
>
> It also makes sense to me to do this as soon as possible (probably once the
> release-1.15 branch is cut) with the example of ElasticSearch. Afterwards
> (hopefully still in the Flink 1.16 release cycle) we do the same for other
> connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or
> helpful to make it a condition that this happens for all connectors at the
> same time.
>
> Cheers and thanks,
>
> Konstantin
>
>
> On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ch...@apache.org> wrote:
>
>> If we don't make a release, I think it would appear as partially
>> externalized (since the binaries are still only created with Flink core,
>> not from the external repository).
>>
>> I'm wondering you are referring to when you say "it appear[s]". Users
>> don't know about it, and contributors can be easily informed about the
>> current state. Who's opinion are you worried about?
>>
>> doing a release [means] that we have then completely externalized the
>> connector
>>
>> In my mind the connector is completely externalized once the connector
>> project is complete and can act independently from the core repo. That
>> includes having all the code, working CI and the documentation being
>> integrated into the Flink website. And /then/ we can do a release. I
>> don't see how this could work any other way; how could we possibly argue
>> that the connector is externalized when development on the connector
>> isn't even possible in that repository?
>>
>> There are also other connectors (like Opensearch and I believe RabbitMQ)
>> that will end up straight in their own repositories
>>
>>
>> Which is a bit of a different situation because here the only source of
>> this connector will have been that repository.
>>
>> Would you prefer to remove a connector in a Flink patch release?
>>
>>
>> No. I think I misread your statement; when you said that there "is 1
>> release cycle where the connector both exists in Flink core and the
>> external repo", you are referring to 1.15, correct? (although this
>> should also apply to 1.14 so isn't it 2 releases...?)
>> How I understood it was that we'd keep the connector around until 1.16,
>> which would obviously be terrible.
>>
>> On 19/02/2022 13:30, Martijn Visser wrote:
>>> Hi Chesnay,
>>>
>>> I think the advantage of also doing a release is that we have then
>>> completely externalized the connector. If we don't make a release, I
>> think
>>> it would appear as partially externalized (since the binaries are still
>>> only created with Flink core, not from the external repository). It would
>>> also mean that in our documentation we would still point to the binary
>>> created with the Flink core release.
>>>
>>> There are also other connectors (like Opensearch and I believe RabbitMQ)
>>> that will end up straight in their own repositories. Since we also would
>>> like to document those, I don't think the situation will be messy. We can
>>> also solve it with an information hint in the documentation.
>>>
>>> With regards to point 6, do you have an alternative? Would you prefer to
>>> remove a connector in a Flink patch release?
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ch...@apache.org>
>> wrote:
>>>> Why do you want to immediately do a release for the elasticsearch
>>>> connector? What does that provide us?
>>>>
>>>> I'd rather first have a fully working setup and integrated documentation
>>>> before thinking about releasing anything.
>>>> Once we have that we may be able to externalize all connectors within 1
>>>> release cycle and do a clean switch; otherwise we end up with a bit of a
>>>> messy situation for users where some connectors use version scheme A and
>>>> others B.
>>>>
>>>> I also doubt the value of 6). They'll have to update the version anyway
>>>> (and discover at some point that the version scheme has changed), so I
>>>> don't see what this makes easier.
>>>>
>>>> On 18/02/2022 14:54, Martijn Visser wrote:
>>>>> Hi everyone,
>>>>>
>>>>> As a follow-up to earlier discussions [1] [2] to externalize the
>>>> connectors
>>>>> from the Flink repository, I would like to propose a plan to
>> externalize
>>>>> these connectors. The goal of this plan is to start with moving
>>>> connectors
>>>>> to its own repositories without introducing regressions for connector
>>>>> developers.
>>>>>
>>>>> The plan is as follows:
>>>>>
>>>>> 1. A new repository is requested for a connector.
>>>>> 2. The code for that connector is moved to its individual repository,
>>>>> including the commit history
>>>>> 3. Any first release made for a connector in an external connector
>>>>> repository starts with major version 3, so 3.0.0. The reason for that
>> is
>>>>> that we want to decouple the Flink releases from a connector release.
>> If
>>>> we
>>>>> would start with major version 2, it could cause some confusion because
>>>>> people could think a Flink 2.0 has been released. This does mean that
>>>> each
>>>>> connector needs to have a compatibility matrix generated, stating which
>>>>> version number of the connector is compatible with the correct Flink
>>>>> version.
>>>>> 4. The group id and artifact id for the connector will remain the same,
>>>>> only the version is different.
>>>>> 5. The connector dependencies on the Flink website are updated to point
>>>> to
>>>>> the newly released connector artifact.
>>>>> 6. If a connector is moved, there is one release cycle where there will
>>>> be
>>>>> binary releases for that connector in both Flink core and from the
>>>>> connector repository. This is to make Flink users who are upgrading
>>>>> slightly easier. We will have to make a note in the release notes that
>> a
>>>>> connector has been moved and that a user should update any references
>>>> from
>>>>> the original connector artifact (from the Flink connector) to the new
>>>>> connector artifact (from the external conenctor version)
>>>>>
>>>>> We propose to first try to execute this plan for the Elasticsearch
>>>>> connector as follows:
>>>>>
>>>>> 1. We wait until the Flink 1.15 release branch is cut
>>>>> 2. When that's done, the Elasticsearch code (including commit history)
>>>> from
>>>>> Flink's 1.15 release branch will be moved to the
>>>>> flink-connector-elasticsearch main branch.
>>>>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
>>>>> connector for the external connector repository with version 3.0.0.
>>>>> 4. Bugfixes or improvements will be made first pointing to the external
>>>>> connector repository and will be cherry-picked back to the release-1.15
>>>>> branch in the Flink core repository.
>>>>> 5. The Elasticsearch code, test etc will be removed from the master
>>>> branch
>>>>> in the Flink core repository and dropped with Flink 1.16
>>>>>
>>>>> Looking forward to your thoughts on this!
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn Visser
>>>>> https://twitter.com/MartijnVisser82
>>>>>
>>>>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>>>>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>>>>>
>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Konstantin Knauf <kn...@apache.org>.

Hi Chesnay, Hi everyone,

I think the idea for the migration is the following (with the example of
ElasticSearch). I talked to Martijn offline.

1. ElasticSearch Connector is released from the core repository with the
Flink 1.15.0 release. No changes.

2. At the beginning of the Flink 1.16 release cycle the connector is
removed from `master` of the core repository. It remains on the
`release-1.15` branch and earlier release branches.

3. The connector code is "copied" over to the `master` branch of a
`flink-connector-elastic-search` repository. Bugfixes to the connector need
to go to both `release-1.15` and before in the core repository and `master`
of the external repository.

4. Once all the processes required to do a release in the
`flink-connector-elastic-search` are in place (docs integration, release
automation,...), we release flink-connector-elastic-search:3.0.0, which
will be compatible with Flink 1.15. At this point, users can choose whether
they use flink-connector-elastic-search:1.15.x (released from the core
repository) or flink-connector-elastic-search:3.0.0 already released from
the external repository with Flink 1.15. The documentation will already
advertise the one released from the external repository. This is the
"overlap" that Martijn mentioned.

5. From here onwards, the release cycle of the ElasticSearch Connector is
independent. There could be 3.1.0 and 3.0.1 etc. The compatibility matrix
will be part of the connector documentation.

6. If there is a patch release for Flink 1.15-, this will of course also
include flink-connector-elastic-search release from the core repository.

7. For Flink 1.16, there might or might not be a release of the
elastic-search-connector from the external repository. Depends on
compatibility.

I hope this clarifies it a bit and it makes sense to me.

It also makes sense to me to do this as soon as possible (probably once the
release-1.15 branch is cut) with the example of ElasticSearch. Afterwards
(hopefully still in the Flink 1.16 release cycle) we do the same for other
connectors like Kafka, Pulsar, Kinesis. I don't think it's feasible or
helpful to make it a condition that this happens for all connectors at the
same time.

Cheers and thanks,

Konstantin


On Sun, Feb 20, 2022 at 7:57 PM Chesnay Schepler <ch...@apache.org> wrote:

> If we don't make a release, I think it would appear as partially
> externalized (since the binaries are still only created with Flink core,
> not from the external repository).
>
> I'm wondering you are referring to when you say "it appear[s]". Users
> don't know about it, and contributors can be easily informed about the
> current state. Who's opinion are you worried about?
>
> doing a release [means] that we have then completely externalized the
> connector
>
> In my mind the connector is completely externalized once the connector
> project is complete and can act independently from the core repo. That
> includes having all the code, working CI and the documentation being
> integrated into the Flink website. And /then/ we can do a release. I
> don't see how this could work any other way; how could we possibly argue
> that the connector is externalized when development on the connector
> isn't even possible in that repository?
>
> There are also other connectors (like Opensearch and I believe RabbitMQ)
> that will end up straight in their own repositories
>
>
> Which is a bit of a different situation because here the only source of
> this connector will have been that repository.
>
> Would you prefer to remove a connector in a Flink patch release?
>
>
> No. I think I misread your statement; when you said that there "is 1
> release cycle where the connector both exists in Flink core and the
> external repo", you are referring to 1.15, correct? (although this
> should also apply to 1.14 so isn't it 2 releases...?)
> How I understood it was that we'd keep the connector around until 1.16,
> which would obviously be terrible.
>
> On 19/02/2022 13:30, Martijn Visser wrote:
> > Hi Chesnay,
> >
> > I think the advantage of also doing a release is that we have then
> > completely externalized the connector. If we don't make a release, I
> think
> > it would appear as partially externalized (since the binaries are still
> > only created with Flink core, not from the external repository). It would
> > also mean that in our documentation we would still point to the binary
> > created with the Flink core release.
> >
> > There are also other connectors (like Opensearch and I believe RabbitMQ)
> > that will end up straight in their own repositories. Since we also would
> > like to document those, I don't think the situation will be messy. We can
> > also solve it with an information hint in the documentation.
> >
> > With regards to point 6, do you have an alternative? Would you prefer to
> > remove a connector in a Flink patch release?
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ch...@apache.org>
> wrote:
> >
> >> Why do you want to immediately do a release for the elasticsearch
> >> connector? What does that provide us?
> >>
> >> I'd rather first have a fully working setup and integrated documentation
> >> before thinking about releasing anything.
> >> Once we have that we may be able to externalize all connectors within 1
> >> release cycle and do a clean switch; otherwise we end up with a bit of a
> >> messy situation for users where some connectors use version scheme A and
> >> others B.
> >>
> >> I also doubt the value of 6). They'll have to update the version anyway
> >> (and discover at some point that the version scheme has changed), so I
> >> don't see what this makes easier.
> >>
> >> On 18/02/2022 14:54, Martijn Visser wrote:
> >>> Hi everyone,
> >>>
> >>> As a follow-up to earlier discussions [1] [2] to externalize the
> >> connectors
> >>> from the Flink repository, I would like to propose a plan to
> externalize
> >>> these connectors. The goal of this plan is to start with moving
> >> connectors
> >>> to its own repositories without introducing regressions for connector
> >>> developers.
> >>>
> >>> The plan is as follows:
> >>>
> >>> 1. A new repository is requested for a connector.
> >>> 2. The code for that connector is moved to its individual repository,
> >>> including the commit history
> >>> 3. Any first release made for a connector in an external connector
> >>> repository starts with major version 3, so 3.0.0. The reason for that
> is
> >>> that we want to decouple the Flink releases from a connector release.
> If
> >> we
> >>> would start with major version 2, it could cause some confusion because
> >>> people could think a Flink 2.0 has been released. This does mean that
> >> each
> >>> connector needs to have a compatibility matrix generated, stating which
> >>> version number of the connector is compatible with the correct Flink
> >>> version.
> >>> 4. The group id and artifact id for the connector will remain the same,
> >>> only the version is different.
> >>> 5. The connector dependencies on the Flink website are updated to point
> >> to
> >>> the newly released connector artifact.
> >>> 6. If a connector is moved, there is one release cycle where there will
> >> be
> >>> binary releases for that connector in both Flink core and from the
> >>> connector repository. This is to make Flink users who are upgrading
> >>> slightly easier. We will have to make a note in the release notes that
> a
> >>> connector has been moved and that a user should update any references
> >> from
> >>> the original connector artifact (from the Flink connector) to the new
> >>> connector artifact (from the external conenctor version)
> >>>
> >>> We propose to first try to execute this plan for the Elasticsearch
> >>> connector as follows:
> >>>
> >>> 1. We wait until the Flink 1.15 release branch is cut
> >>> 2. When that's done, the Elasticsearch code (including commit history)
> >> from
> >>> Flink's 1.15 release branch will be moved to the
> >>> flink-connector-elasticsearch main branch.
> >>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
> >>> connector for the external connector repository with version 3.0.0.
> >>> 4. Bugfixes or improvements will be made first pointing to the external
> >>> connector repository and will be cherry-picked back to the release-1.15
> >>> branch in the Flink core repository.
> >>> 5. The Elasticsearch code, test etc will be removed from the master
> >> branch
> >>> in the Flink core repository and dropped with Flink 1.16
> >>>
> >>> Looking forward to your thoughts on this!
> >>>
> >>> Best regards,
> >>>
> >>> Martijn Visser
> >>> https://twitter.com/MartijnVisser82
> >>>
> >>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> >>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
> >>>
> >>
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Chesnay Schepler <ch...@apache.org>.

If we don't make a release, I think it would appear as partially 
externalized (since the binaries are still only created with Flink core, 
not from the external repository).

I'm wondering you are referring to when you say "it appear[s]". Users 
don't know about it, and contributors can be easily informed about the 
current state. Who's opinion are you worried about?

doing a release [means] that we have then completely externalized the 
connector

In my mind the connector is completely externalized once the connector 
project is complete and can act independently from the core repo. That 
includes having all the code, working CI and the documentation being 
integrated into the Flink website. And /then/ we can do a release. I 
don't see how this could work any other way; how could we possibly argue 
that the connector is externalized when development on the connector 
isn't even possible in that repository?

There are also other connectors (like Opensearch and I believe RabbitMQ) 
that will end up straight in their own repositories


Which is a bit of a different situation because here the only source of 
this connector will have been that repository.

Would you prefer to remove a connector in a Flink patch release?


No. I think I misread your statement; when you said that there "is 1 
release cycle where the connector both exists in Flink core and the 
external repo", you are referring to 1.15, correct? (although this 
should also apply to 1.14 so isn't it 2 releases...?)
How I understood it was that we'd keep the connector around until 1.16, 
which would obviously be terrible.

On 19/02/2022 13:30, Martijn Visser wrote:
> Hi Chesnay,
>
> I think the advantage of also doing a release is that we have then
> completely externalized the connector. If we don't make a release, I think
> it would appear as partially externalized (since the binaries are still
> only created with Flink core, not from the external repository). It would
> also mean that in our documentation we would still point to the binary
> created with the Flink core release.
>
> There are also other connectors (like Opensearch and I believe RabbitMQ)
> that will end up straight in their own repositories. Since we also would
> like to document those, I don't think the situation will be messy. We can
> also solve it with an information hint in the documentation.
>
> With regards to point 6, do you have an alternative? Would you prefer to
> remove a connector in a Flink patch release?
>
> Best regards,
>
> Martijn
>
> On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler<ch...@apache.org>  wrote:
>
>> Why do you want to immediately do a release for the elasticsearch
>> connector? What does that provide us?
>>
>> I'd rather first have a fully working setup and integrated documentation
>> before thinking about releasing anything.
>> Once we have that we may be able to externalize all connectors within 1
>> release cycle and do a clean switch; otherwise we end up with a bit of a
>> messy situation for users where some connectors use version scheme A and
>> others B.
>>
>> I also doubt the value of 6). They'll have to update the version anyway
>> (and discover at some point that the version scheme has changed), so I
>> don't see what this makes easier.
>>
>> On 18/02/2022 14:54, Martijn Visser wrote:
>>> Hi everyone,
>>>
>>> As a follow-up to earlier discussions [1] [2] to externalize the
>> connectors
>>> from the Flink repository, I would like to propose a plan to externalize
>>> these connectors. The goal of this plan is to start with moving
>> connectors
>>> to its own repositories without introducing regressions for connector
>>> developers.
>>>
>>> The plan is as follows:
>>>
>>> 1. A new repository is requested for a connector.
>>> 2. The code for that connector is moved to its individual repository,
>>> including the commit history
>>> 3. Any first release made for a connector in an external connector
>>> repository starts with major version 3, so 3.0.0. The reason for that is
>>> that we want to decouple the Flink releases from a connector release. If
>> we
>>> would start with major version 2, it could cause some confusion because
>>> people could think a Flink 2.0 has been released. This does mean that
>> each
>>> connector needs to have a compatibility matrix generated, stating which
>>> version number of the connector is compatible with the correct Flink
>>> version.
>>> 4. The group id and artifact id for the connector will remain the same,
>>> only the version is different.
>>> 5. The connector dependencies on the Flink website are updated to point
>> to
>>> the newly released connector artifact.
>>> 6. If a connector is moved, there is one release cycle where there will
>> be
>>> binary releases for that connector in both Flink core and from the
>>> connector repository. This is to make Flink users who are upgrading
>>> slightly easier. We will have to make a note in the release notes that a
>>> connector has been moved and that a user should update any references
>> from
>>> the original connector artifact (from the Flink connector) to the new
>>> connector artifact (from the external conenctor version)
>>>
>>> We propose to first try to execute this plan for the Elasticsearch
>>> connector as follows:
>>>
>>> 1. We wait until the Flink 1.15 release branch is cut
>>> 2. When that's done, the Elasticsearch code (including commit history)
>> from
>>> Flink's 1.15 release branch will be moved to the
>>> flink-connector-elasticsearch main branch.
>>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
>>> connector for the external connector repository with version 3.0.0.
>>> 4. Bugfixes or improvements will be made first pointing to the external
>>> connector repository and will be cherry-picked back to the release-1.15
>>> branch in the Flink core repository.
>>> 5. The Elasticsearch code, test etc will be removed from the master
>> branch
>>> in the Flink core repository and dropped with Flink 1.16
>>>
>>> Looking forward to your thoughts on this!
>>>
>>> Best regards,
>>>
>>> Martijn Visser
>>> https://twitter.com/MartijnVisser82
>>>
>>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>>>
>>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Martijn Visser <ma...@ververica.com>.

Hi Chesnay,

I think the advantage of also doing a release is that we have then
completely externalized the connector. If we don't make a release, I think
it would appear as partially externalized (since the binaries are still
only created with Flink core, not from the external repository). It would
also mean that in our documentation we would still point to the binary
created with the Flink core release.

There are also other connectors (like Opensearch and I believe RabbitMQ)
that will end up straight in their own repositories. Since we also would
like to document those, I don't think the situation will be messy. We can
also solve it with an information hint in the documentation.

With regards to point 6, do you have an alternative? Would you prefer to
remove a connector in a Flink patch release?

Best regards,

Martijn

On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler <ch...@apache.org> wrote:

> Why do you want to immediately do a release for the elasticsearch
> connector? What does that provide us?
>
> I'd rather first have a fully working setup and integrated documentation
> before thinking about releasing anything.
> Once we have that we may be able to externalize all connectors within 1
> release cycle and do a clean switch; otherwise we end up with a bit of a
> messy situation for users where some connectors use version scheme A and
> others B.
>
> I also doubt the value of 6). They'll have to update the version anyway
> (and discover at some point that the version scheme has changed), so I
> don't see what this makes easier.
>
> On 18/02/2022 14:54, Martijn Visser wrote:
> > Hi everyone,
> >
> > As a follow-up to earlier discussions [1] [2] to externalize the
> connectors
> > from the Flink repository, I would like to propose a plan to externalize
> > these connectors. The goal of this plan is to start with moving
> connectors
> > to its own repositories without introducing regressions for connector
> > developers.
> >
> > The plan is as follows:
> >
> > 1. A new repository is requested for a connector.
> > 2. The code for that connector is moved to its individual repository,
> > including the commit history
> > 3. Any first release made for a connector in an external connector
> > repository starts with major version 3, so 3.0.0. The reason for that is
> > that we want to decouple the Flink releases from a connector release. If
> we
> > would start with major version 2, it could cause some confusion because
> > people could think a Flink 2.0 has been released. This does mean that
> each
> > connector needs to have a compatibility matrix generated, stating which
> > version number of the connector is compatible with the correct Flink
> > version.
> > 4. The group id and artifact id for the connector will remain the same,
> > only the version is different.
> > 5. The connector dependencies on the Flink website are updated to point
> to
> > the newly released connector artifact.
> > 6. If a connector is moved, there is one release cycle where there will
> be
> > binary releases for that connector in both Flink core and from the
> > connector repository. This is to make Flink users who are upgrading
> > slightly easier. We will have to make a note in the release notes that a
> > connector has been moved and that a user should update any references
> from
> > the original connector artifact (from the Flink connector) to the new
> > connector artifact (from the external conenctor version)
> >
> > We propose to first try to execute this plan for the Elasticsearch
> > connector as follows:
> >
> > 1. We wait until the Flink 1.15 release branch is cut
> > 2. When that's done, the Elasticsearch code (including commit history)
> from
> > Flink's 1.15 release branch will be moved to the
> > flink-connector-elasticsearch main branch.
> > 3. When Flink 1.15 is released, we will also release an Elasticsearch
> > connector for the external connector repository with version 3.0.0.
> > 4. Bugfixes or improvements will be made first pointing to the external
> > connector repository and will be cherry-picked back to the release-1.15
> > branch in the Flink core repository.
> > 5. The Elasticsearch code, test etc will be removed from the master
> branch
> > in the Flink core repository and dropped with Flink 1.16
> >
> > Looking forward to your thoughts on this!
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> >
> > [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> > [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
> >
>
>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Chesnay Schepler <ch...@apache.org>.

Why do you want to immediately do a release for the elasticsearch 
connector? What does that provide us?

I'd rather first have a fully working setup and integrated documentation 
before thinking about releasing anything.
Once we have that we may be able to externalize all connectors within 1 
release cycle and do a clean switch; otherwise we end up with a bit of a 
messy situation for users where some connectors use version scheme A and 
others B.

I also doubt the value of 6). They'll have to update the version anyway 
(and discover at some point that the version scheme has changed), so I 
don't see what this makes easier.

On 18/02/2022 14:54, Martijn Visser wrote:
> Hi everyone,
>
> As a follow-up to earlier discussions [1] [2] to externalize the connectors
> from the Flink repository, I would like to propose a plan to externalize
> these connectors. The goal of this plan is to start with moving connectors
> to its own repositories without introducing regressions for connector
> developers.
>
> The plan is as follows:
>
> 1. A new repository is requested for a connector.
> 2. The code for that connector is moved to its individual repository,
> including the commit history
> 3. Any first release made for a connector in an external connector
> repository starts with major version 3, so 3.0.0. The reason for that is
> that we want to decouple the Flink releases from a connector release. If we
> would start with major version 2, it could cause some confusion because
> people could think a Flink 2.0 has been released. This does mean that each
> connector needs to have a compatibility matrix generated, stating which
> version number of the connector is compatible with the correct Flink
> version.
> 4. The group id and artifact id for the connector will remain the same,
> only the version is different.
> 5. The connector dependencies on the Flink website are updated to point to
> the newly released connector artifact.
> 6. If a connector is moved, there is one release cycle where there will be
> binary releases for that connector in both Flink core and from the
> connector repository. This is to make Flink users who are upgrading
> slightly easier. We will have to make a note in the release notes that a
> connector has been moved and that a user should update any references from
> the original connector artifact (from the Flink connector) to the new
> connector artifact (from the external conenctor version)
>
> We propose to first try to execute this plan for the Elasticsearch
> connector as follows:
>
> 1. We wait until the Flink 1.15 release branch is cut
> 2. When that's done, the Elasticsearch code (including commit history) from
> Flink's 1.15 release branch will be moved to the
> flink-connector-elasticsearch main branch.
> 3. When Flink 1.15 is released, we will also release an Elasticsearch
> connector for the external connector repository with version 3.0.0.
> 4. Bugfixes or improvements will be made first pointing to the external
> connector repository and will be cherry-picked back to the release-1.15
> branch in the Flink core repository.
> 5. The Elasticsearch code, test etc will be removed from the master branch
> in the Flink core repository and dropped with Flink 1.16
>
> Looking forward to your thoughts on this!
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Johannes Moser <jo...@ververica.com>.

+1 (non-binding)

> On 18.02.2022, at 15:12, Konstantin Knauf <kn...@apache.org> wrote:
> 
> +1 to the approach.
> 
> I expect that we will encounter more questions and challenges as we go,
> but these are best discussed and addressed in the context of a specific
> connector like ElasticSearch.
> 
> On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser <ma...@ververica.com>
> wrote:
> 
>> Hi everyone,
>> 
>> As a follow-up to earlier discussions [1] [2] to externalize the connectors
>> from the Flink repository, I would like to propose a plan to externalize
>> these connectors. The goal of this plan is to start with moving connectors
>> to its own repositories without introducing regressions for connector
>> developers.
>> 
>> The plan is as follows:
>> 
>> 1. A new repository is requested for a connector.
>> 2. The code for that connector is moved to its individual repository,
>> including the commit history
>> 3. Any first release made for a connector in an external connector
>> repository starts with major version 3, so 3.0.0. The reason for that is
>> that we want to decouple the Flink releases from a connector release. If we
>> would start with major version 2, it could cause some confusion because
>> people could think a Flink 2.0 has been released. This does mean that each
>> connector needs to have a compatibility matrix generated, stating which
>> version number of the connector is compatible with the correct Flink
>> version.
>> 4. The group id and artifact id for the connector will remain the same,
>> only the version is different.
>> 5. The connector dependencies on the Flink website are updated to point to
>> the newly released connector artifact.
>> 6. If a connector is moved, there is one release cycle where there will be
>> binary releases for that connector in both Flink core and from the
>> connector repository. This is to make Flink users who are upgrading
>> slightly easier. We will have to make a note in the release notes that a
>> connector has been moved and that a user should update any references from
>> the original connector artifact (from the Flink connector) to the new
>> connector artifact (from the external conenctor version)
>> 
>> We propose to first try to execute this plan for the Elasticsearch
>> connector as follows:
>> 
>> 1. We wait until the Flink 1.15 release branch is cut
>> 2. When that's done, the Elasticsearch code (including commit history) from
>> Flink's 1.15 release branch will be moved to the
>> flink-connector-elasticsearch main branch.
>> 3. When Flink 1.15 is released, we will also release an Elasticsearch
>> connector for the external connector repository with version 3.0.0.
>> 4. Bugfixes or improvements will be made first pointing to the external
>> connector repository and will be cherry-picked back to the release-1.15
>> branch in the Flink core repository.
>> 5. The Elasticsearch code, test etc will be removed from the master branch
>> in the Flink core repository and dropped with Flink 1.16
>> 
>> Looking forward to your thoughts on this!
>> 
>> Best regards,
>> 
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>> 
>> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>> [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>> 
> 
> 
> -- 
> 
> Konstantin Knauf
> 
> https://twitter.com/snntrable
> 
> https://github.com/knaufk

Re: [DISCUSS] Plan to externalize connectors and versioning

Posted by Konstantin Knauf <kn...@apache.org>.

+1 to the approach.

 I expect that we will encounter more questions and challenges as we go,
but these are best discussed and addressed in the context of a specific
connector like ElasticSearch.

On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser <ma...@ververica.com>
wrote:

> Hi everyone,
>
> As a follow-up to earlier discussions [1] [2] to externalize the connectors
> from the Flink repository, I would like to propose a plan to externalize
> these connectors. The goal of this plan is to start with moving connectors
> to its own repositories without introducing regressions for connector
> developers.
>
> The plan is as follows:
>
> 1. A new repository is requested for a connector.
> 2. The code for that connector is moved to its individual repository,
> including the commit history
> 3. Any first release made for a connector in an external connector
> repository starts with major version 3, so 3.0.0. The reason for that is
> that we want to decouple the Flink releases from a connector release. If we
> would start with major version 2, it could cause some confusion because
> people could think a Flink 2.0 has been released. This does mean that each
> connector needs to have a compatibility matrix generated, stating which
> version number of the connector is compatible with the correct Flink
> version.
> 4. The group id and artifact id for the connector will remain the same,
> only the version is different.
> 5. The connector dependencies on the Flink website are updated to point to
> the newly released connector artifact.
> 6. If a connector is moved, there is one release cycle where there will be
> binary releases for that connector in both Flink core and from the
> connector repository. This is to make Flink users who are upgrading
> slightly easier. We will have to make a note in the release notes that a
> connector has been moved and that a user should update any references from
> the original connector artifact (from the Flink connector) to the new
> connector artifact (from the external conenctor version)
>
> We propose to first try to execute this plan for the Elasticsearch
> connector as follows:
>
> 1. We wait until the Flink 1.15 release branch is cut
> 2. When that's done, the Elasticsearch code (including commit history) from
> Flink's 1.15 release branch will be moved to the
> flink-connector-elasticsearch main branch.
> 3. When Flink 1.15 is released, we will also release an Elasticsearch
> connector for the external connector repository with version 3.0.0.
> 4. Bugfixes or improvements will be made first pointing to the external
> connector repository and will be cherry-picked back to the release-1.15
> branch in the Flink core repository.
> 5. The Elasticsearch code, test etc will be removed from the master branch
> in the Flink core repository and dropped with Flink 1.16
>
> Looking forward to your thoughts on this!
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk