You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Danny Cranmer <da...@apache.org> on 2022/10/20 20:38:05 UTC
[DISCUSS] Create a dedicated aws-base connector repository
Hello all,
Currently we have 2 AWS Flink connectors in the main Flink codebase
(Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
connector in progress (DynamoDB). Currently all three of these use common
AWS utilities from the flink-connector-aws-base module. Common code
includes client builders, property keys, validation, utils etc.
Once we externalize the connectors, leaving flink-connector-aws-base in the
main Flink repository will restrict our ability to evolve the connectors
quickly. For example, as part of the DynamoDB connector build we are
considering adding a general retry strategy config that can be leveraged by
all connectors. We would need to block on Flink 1.17 for this.
In the past we have tried to keep the AWS SDK version consistent across
connectors, with the externalization this is more likely to diverge.
Option 1: I propose we create a new repository, flink-connector-aws, which
we can move the flink-connector-aws-base module to and create a new
flink-connector-aws-parent to manage SDK versions. Each of the externalized
AWS connectors will depend on this new module and parent. Downside is an
additional module to release per Flink version, however I will volunteer to
manage this.
Option 2: We can move the flink-connector-aws-base module and create
flink-connector-parent within the flink-connector-shared-utils repo [2]
Option 3: We do nothing.
For option 1+2 we will follow the general externalized connector versioning
strategy and rules.
I am inclined towards option 1, and appreciate feedback from the community.
[1]
https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
[2] https://github.com/apache/flink-connector-shared-utils
Thanks,
Danny
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Danny Cranmer <da...@apache.org>.
Thanks all for the feedback. +1 on the single repo and version for AWS
connectors. The reduced maintenance cost and complexity is a clear winner
here.
I will open a vote thread for this matter.
Thanks all!
On Tue, 25 Oct 2022, 03:04 Jark Wu, <im...@gmail.com> wrote:
> TBH, I suspect the way of “a single repository per connector”, considering
> there are hundreds of connectors out there (Airbyte[1], Kafka[2]).
> I don’t think the community is feasible to maintain hundreds of
> repositories.
> It makes sense to combine some connectors to reduce the maintenance
> burden.
> I can imagine we would have a flink-jdbc-connector repo in the future to
> support PG, MySQL, MS SqlServer, Oracle, etc., together.
>
> Best,
> Jark
>
> [1]: https://airbyte.com/connectors
> [2]: https://www.confluent.io/product/connectors/ <
> https://www.confluent.io/product/connectors/>
>
> > 2022年10月25日 06:56，Thomas Weise <th...@apache.org> 写道：
> >
> > Hi Danny,
> >
> > I'm also leaning slightly towards the single AWS connector repo
> direction.
> >
> > Bumps in the underlying AWS SDK would bump all of the connectors in any
> > case. And if a change occurs that is isolated to a single connector, then
> > those that do not use that connector can just skip the release.
> >
> > Cheers,
> > Thomas
> >
> >
> > On Mon, Oct 24, 2022 at 3:01 PM Teoh, Hong <liangtl@amazon.co.uk.invalid
> >
> > wrote:
> >
> >> I like the single repo with single version idea.
> >>
> >> Pros:
> >> - Better discoverability for connectors for AWS services means a better
> >> experience for Flink users
> >> - Natural placement of AWS-related utils (Credentials, SDK Retry
> strategy)
> >>
> >> Caveats:
> >> - As you mentioned, it is not desirable if we have to evolve the major
> >> version of the connector just for a change in a single connector (e.g.
> >> DynamoDB). However, I think it is reasonable to only evolve the major
> >> version of the AWS connector repo when there are Flink Source/Sink API
> >> upgrades or AWS SDK major upgrades (probably quire rare). Any new
> features
> >> for individual connectors can be collapsed into minor releases.
> >> - An additional callout here is that we should be careful adopting any
> AWS
> >> connectors that don't use the AWS SDK directly (e.g. how the Kinesis
> >> connector used KPL for a long time). In my opinion, any new connectors
> like
> >> that would be better placed in their own repositories, otherwise we will
> >> have a complex mesh of dependencies to manage.
> >>
> >> Regards,
> >> Hong
> >>
> >>
> >>
> >>
> >> On 21/10/2022, 16:59, "Danny Cranmer" <da...@apache.org> wrote:
> >>
> >> CAUTION: This email originated from outside of the organization. Do
> >> not click links or open attachments unless you can confirm the sender
> and
> >> know the content is safe.
> >>
> >>
> >>
> >> Thanks Chesnay for the suggestion, I will investigate this option.
> >>
> >> Related to the single repo idea, I have considered it in the past.
> Are
> >> you
> >> proposing we also use a single version between all connectors? If we
> >> have a
> >> single version then it makes sense to combine them in a single repo,
> if
> >> they are separate versions, then splitting them makes sense. This was
> >> discussed last year more generally [1] and the consensus was "we
> >> ultimately
> >> propose to have a single repository per connector".
> >>
> >> Combining all AWS connectors into a single repo with a single version
> >> is
> >> inline with how the AWS SDK works, therefore AWS users are familiar
> >> with
> >> this approach. However it is frustrating that we would have to
> release
> >> all
> >> connectors to fix a bug or add a feature in one of them. Example: a
> >> user is
> >> using Kinesis Data Streams only (the most popular and mature
> >> connector),
> >> and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a
> >> DynamoDB
> >> change.
> >>
> >> I am torn and will think some more, but it would be great to hear
> other
> >> people's opinions.
> >>
> >> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
> >>
> >> Thanks,
> >> Danny
> >>
> >> On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
> >>
> >>> I agree with Jark. It would be easier for the further development and
> >>> maintenance, if all aws related connectors and the base module are
> >> in the
> >>> same repo. It might make sense to upgrade the
> >> flink-connector-dynamodb to
> >>> flink-connector-aws and move the other modules including the
> >>> flink-connector-aws-base into it. The aws sdk could be managed in
> >>> flink-connector-aws-base. Any future common connector features could
> >> also
> >>> be developed in the base module.
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
> >>>
> >>>> How about creating a new repository flink-connector-aws and merging
> >>>> dynamodb, kinesis firehouse into it?
> >>>> This can reduce the maintenance for complex dependencies and make
> >> the
> >>>> release easy.
> >>>> I think the maintainers of aws-releated connectors are the same
> >> people.
> >>>>
> >>>> Best,
> >>>> Jark
> >>>>
> >>>>> 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
> >>>>>
> >>>>> I would not go with 2); I think it'd just be messy .
> >>>>>
> >>>>> Here's another option:
> >>>>>
> >>>>> Create another repository (aws-connector-base) (following the
> >>>> externalization model), add it as a sub-module to the downstream
> >>>> repositories, and make it part of the release process of said
> >> connector.
> >>>>>
> >>>>> I.e., we never create a release for aws-connector-bose, but
> >> release it
> >>>> as part of the connector.
> >>>>> This main benefit here is that we'd always be able to make
> >> changes to
> >>>> the aws-base code without delaying connector releases.
> >>>>> I would assume that any added overhead due to _technically_
> >> releasing
> >>>> the aws code multiple times to be negligible.
> >>>>>
> >>>>>
> >>>>> On 20/10/2022 22:38, Danny Cranmer wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> Currently we have 2 AWS Flink connectors in the main Flink
> >> codebase
> >>>>>> (Kinesis Data Streams and Kinesis Data Firehose) and one new
> >>>> externalized
> >>>>>> connector in progress (DynamoDB). Currently all three of these
> >> use
> >>>> common
> >>>>>> AWS utilities from the flink-connector-aws-base module. Common
> >> code
> >>>>>> includes client builders, property keys, validation, utils etc.
> >>>>>>
> >>>>>> Once we externalize the connectors, leaving
> >> flink-connector-aws-base
> >>>> in the
> >>>>>> main Flink repository will restrict our ability to evolve the
> >>>> connectors
> >>>>>> quickly. For example, as part of the DynamoDB connector build we
> >> are
> >>>>>> considering adding a general retry strategy config that can be
> >>>> leveraged by
> >>>>>> all connectors. We would need to block on Flink 1.17 for this.
> >>>>>>
> >>>>>> In the past we have tried to keep the AWS SDK version consistent
> >> across
> >>>>>> connectors, with the externalization this is more likely to
> >> diverge.
> >>>>>>
> >>>>>> Option 1: I propose we create a new repository,
> >> flink-connector-aws,
> >>>> which
> >>>>>> we can move the flink-connector-aws-base module to and create a
> >> new
> >>>>>> flink-connector-aws-parent to manage SDK versions. Each of the
> >>>> externalized
> >>>>>> AWS connectors will depend on this new module and parent.
> >> Downside is
> >>>> an
> >>>>>> additional module to release per Flink version, however I will
> >>>> volunteer to
> >>>>>> manage this.
> >>>>>>
> >>>>>> Option 2: We can move the flink-connector-aws-base module and
> >> create
> >>>>>> flink-connector-parent within the flink-connector-shared-utils
> >> repo [2]
> >>>>>>
> >>>>>> Option 3: We do nothing.
> >>>>>>
> >>>>>> For option 1+2 we will follow the general externalized connector
> >>>> versioning
> >>>>>> strategy and rules.
> >>>>>>
> >>>>>> I am inclined towards option 1, and appreciate feedback from the
> >>>> community.
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
> >>
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
> >>>>>> [2] https://github.com/apache/flink-connector-shared-utils
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Danny
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Jark Wu <im...@gmail.com>.
TBH, I suspect the way of “a single repository per connector”, considering
there are hundreds of connectors out there (Airbyte[1], Kafka[2]).
I don’t think the community is feasible to maintain hundreds of repositories.
It makes sense to combine some connectors to reduce the maintenance burden.
I can imagine we would have a flink-jdbc-connector repo in the future to
support PG, MySQL, MS SqlServer, Oracle, etc., together.
Best,
Jark
[1]: https://airbyte.com/connectors
[2]: https://www.confluent.io/product/connectors/ <https://www.confluent.io/product/connectors/>
> 2022年10月25日 06:56，Thomas Weise <th...@apache.org> 写道：
>
> Hi Danny,
>
> I'm also leaning slightly towards the single AWS connector repo direction.
>
> Bumps in the underlying AWS SDK would bump all of the connectors in any
> case. And if a change occurs that is isolated to a single connector, then
> those that do not use that connector can just skip the release.
>
> Cheers,
> Thomas
>
>
> On Mon, Oct 24, 2022 at 3:01 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> wrote:
>
>> I like the single repo with single version idea.
>>
>> Pros:
>> - Better discoverability for connectors for AWS services means a better
>> experience for Flink users
>> - Natural placement of AWS-related utils (Credentials, SDK Retry strategy)
>>
>> Caveats:
>> - As you mentioned, it is not desirable if we have to evolve the major
>> version of the connector just for a change in a single connector (e.g.
>> DynamoDB). However, I think it is reasonable to only evolve the major
>> version of the AWS connector repo when there are Flink Source/Sink API
>> upgrades or AWS SDK major upgrades (probably quire rare). Any new features
>> for individual connectors can be collapsed into minor releases.
>> - An additional callout here is that we should be careful adopting any AWS
>> connectors that don't use the AWS SDK directly (e.g. how the Kinesis
>> connector used KPL for a long time). In my opinion, any new connectors like
>> that would be better placed in their own repositories, otherwise we will
>> have a complex mesh of dependencies to manage.
>>
>> Regards,
>> Hong
>>
>>
>>
>>
>> On 21/10/2022, 16:59, "Danny Cranmer" <da...@apache.org> wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>>
>>
>> Thanks Chesnay for the suggestion, I will investigate this option.
>>
>> Related to the single repo idea, I have considered it in the past. Are
>> you
>> proposing we also use a single version between all connectors? If we
>> have a
>> single version then it makes sense to combine them in a single repo, if
>> they are separate versions, then splitting them makes sense. This was
>> discussed last year more generally [1] and the consensus was "we
>> ultimately
>> propose to have a single repository per connector".
>>
>> Combining all AWS connectors into a single repo with a single version
>> is
>> inline with how the AWS SDK works, therefore AWS users are familiar
>> with
>> this approach. However it is frustrating that we would have to release
>> all
>> connectors to fix a bug or add a feature in one of them. Example: a
>> user is
>> using Kinesis Data Streams only (the most popular and mature
>> connector),
>> and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a
>> DynamoDB
>> change.
>>
>> I am torn and will think some more, but it would be great to hear other
>> people's opinions.
>>
>> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>>
>> Thanks,
>> Danny
>>
>> On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>>
>>> I agree with Jark. It would be easier for the further development and
>>> maintenance, if all aws related connectors and the base module are
>> in the
>>> same repo. It might make sense to upgrade the
>> flink-connector-dynamodb to
>>> flink-connector-aws and move the other modules including the
>>> flink-connector-aws-base into it. The aws sdk could be managed in
>>> flink-connector-aws-base. Any future common connector features could
>> also
>>> be developed in the base module.
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
>>>
>>>> How about creating a new repository flink-connector-aws and merging
>>>> dynamodb, kinesis firehouse into it?
>>>> This can reduce the maintenance for complex dependencies and make
>> the
>>>> release easy.
>>>> I think the maintainers of aws-releated connectors are the same
>> people.
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>>> 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
>>>>>
>>>>> I would not go with 2); I think it'd just be messy .
>>>>>
>>>>> Here's another option:
>>>>>
>>>>> Create another repository (aws-connector-base) (following the
>>>> externalization model), add it as a sub-module to the downstream
>>>> repositories, and make it part of the release process of said
>> connector.
>>>>>
>>>>> I.e., we never create a release for aws-connector-bose, but
>> release it
>>>> as part of the connector.
>>>>> This main benefit here is that we'd always be able to make
>> changes to
>>>> the aws-base code without delaying connector releases.
>>>>> I would assume that any added overhead due to _technically_
>> releasing
>>>> the aws code multiple times to be negligible.
>>>>>
>>>>>
>>>>> On 20/10/2022 22:38, Danny Cranmer wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> Currently we have 2 AWS Flink connectors in the main Flink
>> codebase
>>>>>> (Kinesis Data Streams and Kinesis Data Firehose) and one new
>>>> externalized
>>>>>> connector in progress (DynamoDB). Currently all three of these
>> use
>>>> common
>>>>>> AWS utilities from the flink-connector-aws-base module. Common
>> code
>>>>>> includes client builders, property keys, validation, utils etc.
>>>>>>
>>>>>> Once we externalize the connectors, leaving
>> flink-connector-aws-base
>>>> in the
>>>>>> main Flink repository will restrict our ability to evolve the
>>>> connectors
>>>>>> quickly. For example, as part of the DynamoDB connector build we
>> are
>>>>>> considering adding a general retry strategy config that can be
>>>> leveraged by
>>>>>> all connectors. We would need to block on Flink 1.17 for this.
>>>>>>
>>>>>> In the past we have tried to keep the AWS SDK version consistent
>> across
>>>>>> connectors, with the externalization this is more likely to
>> diverge.
>>>>>>
>>>>>> Option 1: I propose we create a new repository,
>> flink-connector-aws,
>>>> which
>>>>>> we can move the flink-connector-aws-base module to and create a
>> new
>>>>>> flink-connector-aws-parent to manage SDK versions. Each of the
>>>> externalized
>>>>>> AWS connectors will depend on this new module and parent.
>> Downside is
>>>> an
>>>>>> additional module to release per Flink version, however I will
>>>> volunteer to
>>>>>> manage this.
>>>>>>
>>>>>> Option 2: We can move the flink-connector-aws-base module and
>> create
>>>>>> flink-connector-parent within the flink-connector-shared-utils
>> repo [2]
>>>>>>
>>>>>> Option 3: We do nothing.
>>>>>>
>>>>>> For option 1+2 we will follow the general externalized connector
>>>> versioning
>>>>>> strategy and rules.
>>>>>>
>>>>>> I am inclined towards option 1, and appreciate feedback from the
>>>> community.
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>>>>>> [2] https://github.com/apache/flink-connector-shared-utils
>>>>>>
>>>>>> Thanks,
>>>>>> Danny
>>>>>>
>>>>>
>>>>
>>>>
>>
>>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Thomas Weise <th...@apache.org>.
Hi Danny,
I'm also leaning slightly towards the single AWS connector repo direction.
Bumps in the underlying AWS SDK would bump all of the connectors in any
case. And if a change occurs that is isolated to a single connector, then
those that do not use that connector can just skip the release.
Cheers,
Thomas
On Mon, Oct 24, 2022 at 3:01 PM Teoh, Hong <li...@amazon.co.uk.invalid>
wrote:
> I like the single repo with single version idea.
>
> Pros:
> - Better discoverability for connectors for AWS services means a better
> experience for Flink users
> - Natural placement of AWS-related utils (Credentials, SDK Retry strategy)
>
> Caveats:
> - As you mentioned, it is not desirable if we have to evolve the major
> version of the connector just for a change in a single connector (e.g.
> DynamoDB). However, I think it is reasonable to only evolve the major
> version of the AWS connector repo when there are Flink Source/Sink API
> upgrades or AWS SDK major upgrades (probably quire rare). Any new features
> for individual connectors can be collapsed into minor releases.
> - An additional callout here is that we should be careful adopting any AWS
> connectors that don't use the AWS SDK directly (e.g. how the Kinesis
> connector used KPL for a long time). In my opinion, any new connectors like
> that would be better placed in their own repositories, otherwise we will
> have a complex mesh of dependencies to manage.
>
> Regards,
> Hong
>
>
>
>
> On 21/10/2022, 16:59, "Danny Cranmer" <da...@apache.org> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> Thanks Chesnay for the suggestion, I will investigate this option.
>
> Related to the single repo idea, I have considered it in the past. Are
> you
> proposing we also use a single version between all connectors? If we
> have a
> single version then it makes sense to combine them in a single repo, if
> they are separate versions, then splitting them makes sense. This was
> discussed last year more generally [1] and the consensus was "we
> ultimately
> propose to have a single repository per connector".
>
> Combining all AWS connectors into a single repo with a single version
> is
> inline with how the AWS SDK works, therefore AWS users are familiar
> with
> this approach. However it is frustrating that we would have to release
> all
> connectors to fix a bug or add a feature in one of them. Example: a
> user is
> using Kinesis Data Streams only (the most popular and mature
> connector),
> and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a
> DynamoDB
> change.
>
> I am torn and will think some more, but it would be great to hear other
> people's opinions.
>
> [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
>
> Thanks,
> Danny
>
> On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
>
> > I agree with Jark. It would be easier for the further development and
> > maintenance, if all aws related connectors and the base module are
> in the
> > same repo. It might make sense to upgrade the
> flink-connector-dynamodb to
> > flink-connector-aws and move the other modules including the
> > flink-connector-aws-base into it. The aws sdk could be managed in
> > flink-connector-aws-base. Any future common connector features could
> also
> > be developed in the base module.
> >
> > Best regards,
> > Jing
> >
> > On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
> >
> >> How about creating a new repository flink-connector-aws and merging
> >> dynamodb, kinesis firehouse into it?
> >> This can reduce the maintenance for complex dependencies and make
> the
> >> release easy.
> >> I think the maintainers of aws-releated connectors are the same
> people.
> >>
> >> Best,
> >> Jark
> >>
> >> > 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
> >> >
> >> > I would not go with 2); I think it'd just be messy .
> >> >
> >> > Here's another option:
> >> >
> >> > Create another repository (aws-connector-base) (following the
> >> externalization model), add it as a sub-module to the downstream
> >> repositories, and make it part of the release process of said
> connector.
> >> >
> >> > I.e., we never create a release for aws-connector-bose, but
> release it
> >> as part of the connector.
> >> > This main benefit here is that we'd always be able to make
> changes to
> >> the aws-base code without delaying connector releases.
> >> > I would assume that any added overhead due to _technically_
> releasing
> >> the aws code multiple times to be negligible.
> >> >
> >> >
> >> > On 20/10/2022 22:38, Danny Cranmer wrote:
> >> >> Hello all,
> >> >>
> >> >> Currently we have 2 AWS Flink connectors in the main Flink
> codebase
> >> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
> >> externalized
> >> >> connector in progress (DynamoDB). Currently all three of these
> use
> >> common
> >> >> AWS utilities from the flink-connector-aws-base module. Common
> code
> >> >> includes client builders, property keys, validation, utils etc.
> >> >>
> >> >> Once we externalize the connectors, leaving
> flink-connector-aws-base
> >> in the
> >> >> main Flink repository will restrict our ability to evolve the
> >> connectors
> >> >> quickly. For example, as part of the DynamoDB connector build we
> are
> >> >> considering adding a general retry strategy config that can be
> >> leveraged by
> >> >> all connectors. We would need to block on Flink 1.17 for this.
> >> >>
> >> >> In the past we have tried to keep the AWS SDK version consistent
> across
> >> >> connectors, with the externalization this is more likely to
> diverge.
> >> >>
> >> >> Option 1: I propose we create a new repository,
> flink-connector-aws,
> >> which
> >> >> we can move the flink-connector-aws-base module to and create a
> new
> >> >> flink-connector-aws-parent to manage SDK versions. Each of the
> >> externalized
> >> >> AWS connectors will depend on this new module and parent.
> Downside is
> >> an
> >> >> additional module to release per Flink version, however I will
> >> volunteer to
> >> >> manage this.
> >> >>
> >> >> Option 2: We can move the flink-connector-aws-base module and
> create
> >> >> flink-connector-parent within the flink-connector-shared-utils
> repo [2]
> >> >>
> >> >> Option 3: We do nothing.
> >> >>
> >> >> For option 1+2 we will follow the general externalized connector
> >> versioning
> >> >> strategy and rules.
> >> >>
> >> >> I am inclined towards option 1, and appreciate feedback from the
> >> community.
> >> >>
> >> >> [1]
> >> >>
> >>
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
> >> >> [2] https://github.com/apache/flink-connector-shared-utils
> >> >>
> >> >> Thanks,
> >> >> Danny
> >> >>
> >> >
> >>
> >>
>
>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
I like the single repo with single version idea.
Pros:
- Better discoverability for connectors for AWS services means a better experience for Flink users
- Natural placement of AWS-related utils (Credentials, SDK Retry strategy)
Caveats:
- As you mentioned, it is not desirable if we have to evolve the major version of the connector just for a change in a single connector (e.g. DynamoDB). However, I think it is reasonable to only evolve the major version of the AWS connector repo when there are Flink Source/Sink API upgrades or AWS SDK major upgrades (probably quire rare). Any new features for individual connectors can be collapsed into minor releases.
- An additional callout here is that we should be careful adopting any AWS connectors that don't use the AWS SDK directly (e.g. how the Kinesis connector used KPL for a long time). In my opinion, any new connectors like that would be better placed in their own repositories, otherwise we will have a complex mesh of dependencies to manage.
Regards,
Hong
On 21/10/2022, 16:59, "Danny Cranmer" <da...@apache.org> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Thanks Chesnay for the suggestion, I will investigate this option.
Related to the single repo idea, I have considered it in the past. Are you
proposing we also use a single version between all connectors? If we have a
single version then it makes sense to combine them in a single repo, if
they are separate versions, then splitting them makes sense. This was
discussed last year more generally [1] and the consensus was "we ultimately
propose to have a single repository per connector".
Combining all AWS connectors into a single repo with a single version is
inline with how the AWS SDK works, therefore AWS users are familiar with
this approach. However it is frustrating that we would have to release all
connectors to fix a bug or add a feature in one of them. Example: a user is
using Kinesis Data Streams only (the most popular and mature connector),
and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a DynamoDB
change.
I am torn and will think some more, but it would be great to hear other
people's opinions.
[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
Thanks,
Danny
On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
> I agree with Jark. It would be easier for the further development and
> maintenance, if all aws related connectors and the base module are in the
> same repo. It might make sense to upgrade the flink-connector-dynamodb to
> flink-connector-aws and move the other modules including the
> flink-connector-aws-base into it. The aws sdk could be managed in
> flink-connector-aws-base. Any future common connector features could also
> be developed in the base module.
>
> Best regards,
> Jing
>
> On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
>
>> How about creating a new repository flink-connector-aws and merging
>> dynamodb, kinesis firehouse into it?
>> This can reduce the maintenance for complex dependencies and make the
>> release easy.
>> I think the maintainers of aws-releated connectors are the same people.
>>
>> Best,
>> Jark
>>
>> > 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
>> >
>> > I would not go with 2); I think it'd just be messy .
>> >
>> > Here's another option:
>> >
>> > Create another repository (aws-connector-base) (following the
>> externalization model), add it as a sub-module to the downstream
>> repositories, and make it part of the release process of said connector.
>> >
>> > I.e., we never create a release for aws-connector-bose, but release it
>> as part of the connector.
>> > This main benefit here is that we'd always be able to make changes to
>> the aws-base code without delaying connector releases.
>> > I would assume that any added overhead due to _technically_ releasing
>> the aws code multiple times to be negligible.
>> >
>> >
>> > On 20/10/2022 22:38, Danny Cranmer wrote:
>> >> Hello all,
>> >>
>> >> Currently we have 2 AWS Flink connectors in the main Flink codebase
>> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
>> externalized
>> >> connector in progress (DynamoDB). Currently all three of these use
>> common
>> >> AWS utilities from the flink-connector-aws-base module. Common code
>> >> includes client builders, property keys, validation, utils etc.
>> >>
>> >> Once we externalize the connectors, leaving flink-connector-aws-base
>> in the
>> >> main Flink repository will restrict our ability to evolve the
>> connectors
>> >> quickly. For example, as part of the DynamoDB connector build we are
>> >> considering adding a general retry strategy config that can be
>> leveraged by
>> >> all connectors. We would need to block on Flink 1.17 for this.
>> >>
>> >> In the past we have tried to keep the AWS SDK version consistent across
>> >> connectors, with the externalization this is more likely to diverge.
>> >>
>> >> Option 1: I propose we create a new repository, flink-connector-aws,
>> which
>> >> we can move the flink-connector-aws-base module to and create a new
>> >> flink-connector-aws-parent to manage SDK versions. Each of the
>> externalized
>> >> AWS connectors will depend on this new module and parent. Downside is
>> an
>> >> additional module to release per Flink version, however I will
>> volunteer to
>> >> manage this.
>> >>
>> >> Option 2: We can move the flink-connector-aws-base module and create
>> >> flink-connector-parent within the flink-connector-shared-utils repo [2]
>> >>
>> >> Option 3: We do nothing.
>> >>
>> >> For option 1+2 we will follow the general externalized connector
>> versioning
>> >> strategy and rules.
>> >>
>> >> I am inclined towards option 1, and appreciate feedback from the
>> community.
>> >>
>> >> [1]
>> >>
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>> >> [2] https://github.com/apache/flink-connector-shared-utils
>> >>
>> >> Thanks,
>> >> Danny
>> >>
>> >
>>
>>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Danny Cranmer <da...@apache.org>.
Thanks Chesnay for the suggestion, I will investigate this option.
Related to the single repo idea, I have considered it in the past. Are you
proposing we also use a single version between all connectors? If we have a
single version then it makes sense to combine them in a single repo, if
they are separate versions, then splitting them makes sense. This was
discussed last year more generally [1] and the consensus was "we ultimately
propose to have a single repository per connector".
Combining all AWS connectors into a single repo with a single version is
inline with how the AWS SDK works, therefore AWS users are familiar with
this approach. However it is frustrating that we would have to release all
connectors to fix a bug or add a feature in one of them. Example: a user is
using Kinesis Data Streams only (the most popular and mature connector),
and we evolve the version from 1.x to 2.y (or 1.x to 1.y) for a DynamoDB
change.
I am torn and will think some more, but it would be great to hear other
people's opinions.
[1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm
Thanks,
Danny
On Fri, Oct 21, 2022 at 3:11 PM Jing Ge <ji...@ververica.com> wrote:
> I agree with Jark. It would be easier for the further development and
> maintenance, if all aws related connectors and the base module are in the
> same repo. It might make sense to upgrade the flink-connector-dynamodb to
> flink-connector-aws and move the other modules including the
> flink-connector-aws-base into it. The aws sdk could be managed in
> flink-connector-aws-base. Any future common connector features could also
> be developed in the base module.
>
> Best regards,
> Jing
>
> On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
>
>> How about creating a new repository flink-connector-aws and merging
>> dynamodb, kinesis firehouse into it?
>> This can reduce the maintenance for complex dependencies and make the
>> release easy.
>> I think the maintainers of aws-releated connectors are the same people.
>>
>> Best,
>> Jark
>>
>> > 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
>> >
>> > I would not go with 2); I think it'd just be messy .
>> >
>> > Here's another option:
>> >
>> > Create another repository (aws-connector-base) (following the
>> externalization model), add it as a sub-module to the downstream
>> repositories, and make it part of the release process of said connector.
>> >
>> > I.e., we never create a release for aws-connector-bose, but release it
>> as part of the connector.
>> > This main benefit here is that we'd always be able to make changes to
>> the aws-base code without delaying connector releases.
>> > I would assume that any added overhead due to _technically_ releasing
>> the aws code multiple times to be negligible.
>> >
>> >
>> > On 20/10/2022 22:38, Danny Cranmer wrote:
>> >> Hello all,
>> >>
>> >> Currently we have 2 AWS Flink connectors in the main Flink codebase
>> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
>> externalized
>> >> connector in progress (DynamoDB). Currently all three of these use
>> common
>> >> AWS utilities from the flink-connector-aws-base module. Common code
>> >> includes client builders, property keys, validation, utils etc.
>> >>
>> >> Once we externalize the connectors, leaving flink-connector-aws-base
>> in the
>> >> main Flink repository will restrict our ability to evolve the
>> connectors
>> >> quickly. For example, as part of the DynamoDB connector build we are
>> >> considering adding a general retry strategy config that can be
>> leveraged by
>> >> all connectors. We would need to block on Flink 1.17 for this.
>> >>
>> >> In the past we have tried to keep the AWS SDK version consistent across
>> >> connectors, with the externalization this is more likely to diverge.
>> >>
>> >> Option 1: I propose we create a new repository, flink-connector-aws,
>> which
>> >> we can move the flink-connector-aws-base module to and create a new
>> >> flink-connector-aws-parent to manage SDK versions. Each of the
>> externalized
>> >> AWS connectors will depend on this new module and parent. Downside is
>> an
>> >> additional module to release per Flink version, however I will
>> volunteer to
>> >> manage this.
>> >>
>> >> Option 2: We can move the flink-connector-aws-base module and create
>> >> flink-connector-parent within the flink-connector-shared-utils repo [2]
>> >>
>> >> Option 3: We do nothing.
>> >>
>> >> For option 1+2 we will follow the general externalized connector
>> versioning
>> >> strategy and rules.
>> >>
>> >> I am inclined towards option 1, and appreciate feedback from the
>> community.
>> >>
>> >> [1]
>> >>
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>> >> [2] https://github.com/apache/flink-connector-shared-utils
>> >>
>> >> Thanks,
>> >> Danny
>> >>
>> >
>>
>>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Jing Ge <ji...@ververica.com>.
I agree with Jark. It would be easier for the further development and
maintenance, if all aws related connectors and the base module are in the
same repo. It might make sense to upgrade the flink-connector-dynamodb to
flink-connector-aws and move the other modules including the
flink-connector-aws-base into it. The aws sdk could be managed in
flink-connector-aws-base. Any future common connector features could also
be developed in the base module.
Best regards,
Jing
On Fri, Oct 21, 2022 at 1:26 PM Jark Wu <im...@gmail.com> wrote:
> How about creating a new repository flink-connector-aws and merging
> dynamodb, kinesis firehouse into it?
> This can reduce the maintenance for complex dependencies and make the
> release easy.
> I think the maintainers of aws-releated connectors are the same people.
>
> Best,
> Jark
>
> > 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
> >
> > I would not go with 2); I think it'd just be messy .
> >
> > Here's another option:
> >
> > Create another repository (aws-connector-base) (following the
> externalization model), add it as a sub-module to the downstream
> repositories, and make it part of the release process of said connector.
> >
> > I.e., we never create a release for aws-connector-bose, but release it
> as part of the connector.
> > This main benefit here is that we'd always be able to make changes to
> the aws-base code without delaying connector releases.
> > I would assume that any added overhead due to _technically_ releasing
> the aws code multiple times to be negligible.
> >
> >
> > On 20/10/2022 22:38, Danny Cranmer wrote:
> >> Hello all,
> >>
> >> Currently we have 2 AWS Flink connectors in the main Flink codebase
> >> (Kinesis Data Streams and Kinesis Data Firehose) and one new
> externalized
> >> connector in progress (DynamoDB). Currently all three of these use
> common
> >> AWS utilities from the flink-connector-aws-base module. Common code
> >> includes client builders, property keys, validation, utils etc.
> >>
> >> Once we externalize the connectors, leaving flink-connector-aws-base in
> the
> >> main Flink repository will restrict our ability to evolve the connectors
> >> quickly. For example, as part of the DynamoDB connector build we are
> >> considering adding a general retry strategy config that can be
> leveraged by
> >> all connectors. We would need to block on Flink 1.17 for this.
> >>
> >> In the past we have tried to keep the AWS SDK version consistent across
> >> connectors, with the externalization this is more likely to diverge.
> >>
> >> Option 1: I propose we create a new repository, flink-connector-aws,
> which
> >> we can move the flink-connector-aws-base module to and create a new
> >> flink-connector-aws-parent to manage SDK versions. Each of the
> externalized
> >> AWS connectors will depend on this new module and parent. Downside is an
> >> additional module to release per Flink version, however I will
> volunteer to
> >> manage this.
> >>
> >> Option 2: We can move the flink-connector-aws-base module and create
> >> flink-connector-parent within the flink-connector-shared-utils repo [2]
> >>
> >> Option 3: We do nothing.
> >>
> >> For option 1+2 we will follow the general externalized connector
> versioning
> >> strategy and rules.
> >>
> >> I am inclined towards option 1, and appreciate feedback from the
> community.
> >>
> >> [1]
> >>
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
> >> [2] https://github.com/apache/flink-connector-shared-utils
> >>
> >> Thanks,
> >> Danny
> >>
> >
>
>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Jark Wu <im...@gmail.com>.
How about creating a new repository flink-connector-aws and merging dynamodb, kinesis firehouse into it?
This can reduce the maintenance for complex dependencies and make the release easy.
I think the maintainers of aws-releated connectors are the same people.
Best,
Jark
> 2022年10月21日 17:41，Chesnay Schepler <ch...@apache.org> 写道：
>
> I would not go with 2); I think it'd just be messy .
>
> Here's another option:
>
> Create another repository (aws-connector-base) (following the externalization model), add it as a sub-module to the downstream repositories, and make it part of the release process of said connector.
>
> I.e., we never create a release for aws-connector-bose, but release it as part of the connector.
> This main benefit here is that we'd always be able to make changes to the aws-base code without delaying connector releases.
> I would assume that any added overhead due to _technically_ releasing the aws code multiple times to be negligible.
>
>
> On 20/10/2022 22:38, Danny Cranmer wrote:
>> Hello all,
>>
>> Currently we have 2 AWS Flink connectors in the main Flink codebase
>> (Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
>> connector in progress (DynamoDB). Currently all three of these use common
>> AWS utilities from the flink-connector-aws-base module. Common code
>> includes client builders, property keys, validation, utils etc.
>>
>> Once we externalize the connectors, leaving flink-connector-aws-base in the
>> main Flink repository will restrict our ability to evolve the connectors
>> quickly. For example, as part of the DynamoDB connector build we are
>> considering adding a general retry strategy config that can be leveraged by
>> all connectors. We would need to block on Flink 1.17 for this.
>>
>> In the past we have tried to keep the AWS SDK version consistent across
>> connectors, with the externalization this is more likely to diverge.
>>
>> Option 1: I propose we create a new repository, flink-connector-aws, which
>> we can move the flink-connector-aws-base module to and create a new
>> flink-connector-aws-parent to manage SDK versions. Each of the externalized
>> AWS connectors will depend on this new module and parent. Downside is an
>> additional module to release per Flink version, however I will volunteer to
>> manage this.
>>
>> Option 2: We can move the flink-connector-aws-base module and create
>> flink-connector-parent within the flink-connector-shared-utils repo [2]
>>
>> Option 3: We do nothing.
>>
>> For option 1+2 we will follow the general externalized connector versioning
>> strategy and rules.
>>
>> I am inclined towards option 1, and appreciate feedback from the community.
>>
>> [1]
>> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
>> [2] https://github.com/apache/flink-connector-shared-utils
>>
>> Thanks,
>> Danny
>>
>
Re: [DISCUSS] Create a dedicated aws-base connector repository
Posted by Chesnay Schepler <ch...@apache.org>.
I would not go with 2); I think it'd just be messy .
Here's another option:
Create another repository (aws-connector-base) (following the
externalization model), add it as a sub-module to the downstream
repositories, and make it part of the release process of said connector.
I.e., we never create a release for aws-connector-bose, but release it
as part of the connector.
This main benefit here is that we'd always be able to make changes to
the aws-base code without delaying connector releases.
I would assume that any added overhead due to _technically_ releasing
the aws code multiple times to be negligible.
On 20/10/2022 22:38, Danny Cranmer wrote:
> Hello all,
>
> Currently we have 2 AWS Flink connectors in the main Flink codebase
> (Kinesis Data Streams and Kinesis Data Firehose) and one new externalized
> connector in progress (DynamoDB). Currently all three of these use common
> AWS utilities from the flink-connector-aws-base module. Common code
> includes client builders, property keys, validation, utils etc.
>
> Once we externalize the connectors, leaving flink-connector-aws-base in the
> main Flink repository will restrict our ability to evolve the connectors
> quickly. For example, as part of the DynamoDB connector build we are
> considering adding a general retry strategy config that can be leveraged by
> all connectors. We would need to block on Flink 1.17 for this.
>
> In the past we have tried to keep the AWS SDK version consistent across
> connectors, with the externalization this is more likely to diverge.
>
> Option 1: I propose we create a new repository, flink-connector-aws, which
> we can move the flink-connector-aws-base module to and create a new
> flink-connector-aws-parent to manage SDK versions. Each of the externalized
> AWS connectors will depend on this new module and parent. Downside is an
> additional module to release per Flink version, however I will volunteer to
> manage this.
>
> Option 2: We can move the flink-connector-aws-base module and create
> flink-connector-parent within the flink-connector-shared-utils repo [2]
>
> Option 3: We do nothing.
>
> For option 1+2 we will follow the general externalized connector versioning
> strategy and rules.
>
> I am inclined towards option 1, and appreciate feedback from the community.
>
> [1]
> https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-aws-base
> [2] https://github.com/apache/flink-connector-shared-utils
>
> Thanks,
> Danny
>