[DISCUSS] Process regarding to new connectors

[Martijn Visser <ma...@apache.org>]

Hi everyone,

I would like to open a discussion on the process regarding new connectors.
As you know from previous updates [1] we are making a lot of progress on
externalizing connectors that are currently hosted inside in the Flink
repository.

One topic I would like to bring up for discussion is how the Flink
community wants to deal with new connectors. I've been contacted by many
contributors who are interested in working on one or more connectors. Most
of these connectors are not yet made public or development hasn't started
yet.

When reading up on the Flink Bylaws [2] I would argue that connectors that
are currently already existing (but not under a Flink project scope) would
fall under 'Adoption of New Codebase' which would require a 2/3 majority
vote by PMC members. Looking at the FLIP requirements [3] you could argue
that any new connector is 'a major new feature, subsystem, or piece of
functionality'. A pro of needing to create a (small) FLIP for a new
connector is that someone needs to think about the design, implementation
and requires a vote, so there is more control. The downside of it is that a
FLIP is considered a drawback, given that a connector normally should be
using only the public interfaces provided by Flink so you could argue it's
just an implementation.

I'm looking forward to your input to see if we can reach consensus on this
topic, so it can be included in the documentation for contributors that
want to work and maintain a new connector.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h
[2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
[3]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
?

Re: [DISCUSS] Process regarding to new connectors

[Jark Wu <im...@gmail.com>]

Hi Martijn,

Thanks for bringing up this discussion.

From my point of view, Flink Bylaws should also be applied to the
connectors.
I don't think connectors are just implementations, they provide many APIs
for
end-users, including DataStream API, and SQL DDL options, SQL metadata
columns.

There are many FLIPs just discussing the connector APIs, e.g. FLIP-86[1],
FLIP-125[2], FLIP-105[3], FLIP-107[4].

Best,
Jark

[1]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-86%3A+Improve+Connector+Properties
[2]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-125%3A+Confluent+Schema+Registry+Catalog
[3]:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427289
[4]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Handling+of+metadata+in+SQL+connectors#FLIP107:HandlingofmetadatainSQLconnectors-Kafka




On Thu, 5 May 2022 at 20:00, Martijn Visser <ma...@apache.org>
wrote:

> Hi everyone,
>
> I would like to open a discussion on the process regarding new connectors.
> As you know from previous updates [1] we are making a lot of progress on
> externalizing connectors that are currently hosted inside in the Flink
> repository.
>
> One topic I would like to bring up for discussion is how the Flink
> community wants to deal with new connectors. I've been contacted by many
> contributors who are interested in working on one or more connectors. Most
> of these connectors are not yet made public or development hasn't started
> yet.
>
> When reading up on the Flink Bylaws [2] I would argue that connectors that
> are currently already existing (but not under a Flink project scope) would
> fall under 'Adoption of New Codebase' which would require a 2/3 majority
> vote by PMC members. Looking at the FLIP requirements [3] you could argue
> that any new connector is 'a major new feature, subsystem, or piece of
> functionality'. A pro of needing to create a (small) FLIP for a new
> connector is that someone needs to think about the design, implementation
> and requires a vote, so there is more control. The downside of it is that a
> FLIP is considered a drawback, given that a connector normally should be
> using only the public interfaces provided by Flink so you could argue it's
> just an implementation.
>
> I'm looking forward to your input to see if we can reach consensus on this
> topic, so it can be included in the documentation for contributors that
> want to work and maintain a new connector.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h
> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> [3]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Whatisconsidereda%22majorchange%22thatneedsaFLIP
> ?
>