You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by tison <wa...@gmail.com> on 2022/07/22 15:55:36 UTC
[DISCUSS] Contribute back modifications of MySQL CDC connector
Hi,
I notice that inlong-sort mysql-cdc connector[1] is almost copied from
Flink CDC mysql connector[2].
With a short discussion with Charles I learned that InLong developers added
metadata supports and migration of a whole database.
It could be better to see if InLong community can contribute such supports
back to Flink CDC upstream, and avoid maintaining a whole fork as well as
periodically catch up upstream updates.
Maintainers of Flink CDC projects are in cc (Leonard & Jark).
Perhaps we can start with creating an issue on Flink CDC repo to describe
the case, contribute back the improvements, and switch dependency to the
upstream.
Best,
tison.
[1]
https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
[2]
https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Zirui Sting <zi...@apache.org>.
Hi all,
I created some issues regarding to the discussion above,
https://github.com/ververica/flink-cdc-connectors/issues/1140
https://github.com/ververica/flink-cdc-connectors/issues/1138
https://github.com/ververica/flink-cdc-connectors/issues/1135
Maybe we can discuss them under these specific issues, thanks a lot !
Yours,
PENG
On Mon, Jul 25, 2022 at 10:50 AM Charles Zhang <do...@apache.org>
wrote:
> Thanks, guys. Let's move forward with the discussion and resolution of
> these issues.
>
>
> Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:36写道：
>
> > Hi, Leonard,
> >
> > I created some issues regarding to the discussion above,
> >
> > https://github.com/ververica/flink-cdc-connectors/issues/1140
> > https://github.com/ververica/flink-cdc-connectors/issues/1138
> > https://github.com/ververica/flink-cdc-connectors/issues/1135
> >
> > Maybe we can discuss them under these specific issues, thanks a lot !
> >
> > Yours,
> > PENG
> >
> > On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > Thanks tison for starting the discussion here.
> > >
> > > Both Apache Inlong and Flink CDC Connectors are under Apache License
> 2.0
> > > protocol, It’s legal even someone copy/distribute all code only if
> he/she
> > > keeps the license files.
> > > But from the open source project’s long term maintaining perspective,
> > > personally I like upstream first way instead of copy/distribute way. I
> > > really hope Flink CDC project can help Apache
> > > InLong more and we’re welcome to open issue to statisfy the user
> > > requirements from inLong community.
> > >
> > >
> > > Best,
> > > Leonard Xu
> > >
> > > > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> > > >
> > > > Hi, tison and Jark,
> > > > Thanks for your advice.
> > > > Different from Flink CDC connectors, InLong's changes to MySQL
> CDC
> > > connector mainly involve entire database migration and support for rich
> > > metadata fields (like canal), which are very important for Apache
> InLong
> > to
> > > provide external services as data integration. For example, for the
> > entire
> > > database migration, if one schema is used for one table, it will be a
> > waste
> > > of resources and difficult to maintain, and it will also put pressure
> on
> > > the upstream database.We also particularly hope to work with Flink CDC
> to
> > > see how to support the entire database migration and support richer
> > > metadata fields, so that we can directly rely on it. We don't know if
> > Flink
> > > CDC has plans and technical solutions for these two scenarios. We can
> > talk
> > > about it together or contribute directly.
> > > > In addition, in order to speed up discussion and follow-up
> > > cooperation, I have created a WeChat group [1], and interested
> > contributors
> > > can join the direct discussion.
> > > > Thanks again.
> > > > [1]
> > > > <image.png>
> > > >
> > > > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> > > 22:45写道：
> > > > Thanks for starting this discussion, tison!
> > > >
> > > > It's great to see InLong is using flink-cdc-connectors to extract
> data
> > > from
> > > > various databases.
> > > > I think tison raised a good point about contributing back instead of
> > > > forking. I believe this can
> > > > help the InLong community in the long term by leveraging improvements
> > and
> > > > bug fixes from
> > > > upstream. Flink CDC can also benefit from more contributions from the
> > > > InLong community.
> > > > I believe it's the only way that we can improve collaboration between
> > > > different OSS projects.
> > > >
> > > > In a word, I highly appreciate learning the missing part of Flink CDC
> > > that
> > > > InLong needs.
> > > > We can discuss the missing features here or in the issues.
> > > >
> > > > Cheers,
> > > > Jark
> > > >
> > > >
> > > > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> > > wander4096@gmail.com>> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I notice that inlong-sort mysql-cdc connector[1] is almost copied
> > from
> > > > > Flink CDC mysql connector[2].
> > > > >
> > > > > With a short discussion with Charles I learned that InLong
> developers
> > > > > added metadata supports and migration of a whole database.
> > > > >
> > > > > It could be better to see if InLong community can contribute such
> > > supports
> > > > > back to Flink CDC upstream, and avoid maintaining a whole fork as
> > well
> > > as
> > > > > periodically catch up upstream updates.
> > > > >
> > > > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> > > > >
> > > > > Perhaps we can start with creating an issue on Flink CDC repo to
> > > describe
> > > > > the case, contribute back the improvements, and switch dependency
> to
> > > the
> > > > > upstream.
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > > [1]
> > > > >
> > >
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > > <
> > >
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > > >
> > > > > [2]
> > > > >
> > >
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> > > <
> > >
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best wishes,
> > > > Charles Zhang
> > >
> > >
> >
>
>
> --
> Best wishes,
> Charles Zhang
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Charles Zhang <do...@apache.org>.
Thanks, guys. Let's move forward with the discussion and resolution of
these issues.
Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:36写道：
> Hi, Leonard,
>
> I created some issues regarding to the discussion above,
>
> https://github.com/ververica/flink-cdc-connectors/issues/1140
> https://github.com/ververica/flink-cdc-connectors/issues/1138
> https://github.com/ververica/flink-cdc-connectors/issues/1135
>
> Maybe we can discuss them under these specific issues, thanks a lot !
>
> Yours,
> PENG
>
> On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
>
> > Thanks tison for starting the discussion here.
> >
> > Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0
> > protocol, It’s legal even someone copy/distribute all code only if he/she
> > keeps the license files.
> > But from the open source project’s long term maintaining perspective,
> > personally I like upstream first way instead of copy/distribute way. I
> > really hope Flink CDC project can help Apache
> > InLong more and we’re welcome to open issue to statisfy the user
> > requirements from inLong community.
> >
> >
> > Best,
> > Leonard Xu
> >
> > > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> > >
> > > Hi, tison and Jark,
> > > Thanks for your advice.
> > > Different from Flink CDC connectors, InLong's changes to MySQL CDC
> > connector mainly involve entire database migration and support for rich
> > metadata fields (like canal), which are very important for Apache InLong
> to
> > provide external services as data integration. For example, for the
> entire
> > database migration, if one schema is used for one table, it will be a
> waste
> > of resources and difficult to maintain, and it will also put pressure on
> > the upstream database.We also particularly hope to work with Flink CDC to
> > see how to support the entire database migration and support richer
> > metadata fields, so that we can directly rely on it. We don't know if
> Flink
> > CDC has plans and technical solutions for these two scenarios. We can
> talk
> > about it together or contribute directly.
> > > In addition, in order to speed up discussion and follow-up
> > cooperation, I have created a WeChat group [1], and interested
> contributors
> > can join the direct discussion.
> > > Thanks again.
> > > [1]
> > > <image.png>
> > >
> > > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> > 22:45写道：
> > > Thanks for starting this discussion, tison!
> > >
> > > It's great to see InLong is using flink-cdc-connectors to extract data
> > from
> > > various databases.
> > > I think tison raised a good point about contributing back instead of
> > > forking. I believe this can
> > > help the InLong community in the long term by leveraging improvements
> and
> > > bug fixes from
> > > upstream. Flink CDC can also benefit from more contributions from the
> > > InLong community.
> > > I believe it's the only way that we can improve collaboration between
> > > different OSS projects.
> > >
> > > In a word, I highly appreciate learning the missing part of Flink CDC
> > that
> > > InLong needs.
> > > We can discuss the missing features here or in the issues.
> > >
> > > Cheers,
> > > Jark
> > >
> > >
> > > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> > wander4096@gmail.com>> wrote:
> > >
> > > > Hi,
> > > >
> > > > I notice that inlong-sort mysql-cdc connector[1] is almost copied
> from
> > > > Flink CDC mysql connector[2].
> > > >
> > > > With a short discussion with Charles I learned that InLong developers
> > > > added metadata supports and migration of a whole database.
> > > >
> > > > It could be better to see if InLong community can contribute such
> > supports
> > > > back to Flink CDC upstream, and avoid maintaining a whole fork as
> well
> > as
> > > > periodically catch up upstream updates.
> > > >
> > > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> > > >
> > > > Perhaps we can start with creating an issue on Flink CDC repo to
> > describe
> > > > the case, contribute back the improvements, and switch dependency to
> > the
> > > > upstream.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > > [1]
> > > >
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > <
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > >
> > > > [2]
> > > >
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> > <
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best wishes,
> > > Charles Zhang
> >
> >
>
--
Best wishes,
Charles Zhang
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Zirui Sting <zi...@apache.org>.
Hi, Leonard,
I created some issues regarding to the discussion above,
https://github.com/ververica/flink-cdc-connectors/issues/1140
https://github.com/ververica/flink-cdc-connectors/issues/1138
https://github.com/ververica/flink-cdc-connectors/issues/1135
Maybe we can discuss them under these specific issues, thanks a lot !
Yours,
PENG
On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
> Thanks tison for starting the discussion here.
>
> Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0
> protocol, It’s legal even someone copy/distribute all code only if he/she
> keeps the license files.
> But from the open source project’s long term maintaining perspective,
> personally I like upstream first way instead of copy/distribute way. I
> really hope Flink CDC project can help Apache
> InLong more and we’re welcome to open issue to statisfy the user
> requirements from inLong community.
>
>
> Best,
> Leonard Xu
>
> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> >
> > Hi, tison and Jark,
> > Thanks for your advice.
> > Different from Flink CDC connectors, InLong's changes to MySQL CDC
> connector mainly involve entire database migration and support for rich
> metadata fields (like canal), which are very important for Apache InLong to
> provide external services as data integration. For example, for the entire
> database migration, if one schema is used for one table, it will be a waste
> of resources and difficult to maintain, and it will also put pressure on
> the upstream database.We also particularly hope to work with Flink CDC to
> see how to support the entire database migration and support richer
> metadata fields, so that we can directly rely on it. We don't know if Flink
> CDC has plans and technical solutions for these two scenarios. We can talk
> about it together or contribute directly.
> > In addition, in order to speed up discussion and follow-up
> cooperation, I have created a WeChat group [1], and interested contributors
> can join the direct discussion.
> > Thanks again.
> > [1]
> > <image.png>
> >
> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> 22:45写道：
> > Thanks for starting this discussion, tison!
> >
> > It's great to see InLong is using flink-cdc-connectors to extract data
> from
> > various databases.
> > I think tison raised a good point about contributing back instead of
> > forking. I believe this can
> > help the InLong community in the long term by leveraging improvements and
> > bug fixes from
> > upstream. Flink CDC can also benefit from more contributions from the
> > InLong community.
> > I believe it's the only way that we can improve collaboration between
> > different OSS projects.
> >
> > In a word, I highly appreciate learning the missing part of Flink CDC
> that
> > InLong needs.
> > We can discuss the missing features here or in the issues.
> >
> > Cheers,
> > Jark
> >
> >
> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> wander4096@gmail.com>> wrote:
> >
> > > Hi,
> > >
> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> > > Flink CDC mysql connector[2].
> > >
> > > With a short discussion with Charles I learned that InLong developers
> > > added metadata supports and migration of a whole database.
> > >
> > > It could be better to see if InLong community can contribute such
> supports
> > > back to Flink CDC upstream, and avoid maintaining a whole fork as well
> as
> > > periodically catch up upstream updates.
> > >
> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> > >
> > > Perhaps we can start with creating an issue on Flink CDC repo to
> describe
> > > the case, contribute back the improvements, and switch dependency to
> the
> > > upstream.
> > >
> > > Best,
> > > tison.
> > >
> > > [1]
> > >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> <
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >
> > > [2]
> > >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> <
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >
> > >
> > >
> >
> >
> > --
> > Best wishes,
> > Charles Zhang
>
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Goson zhang <go...@apache.org>.
I found that although issues have been pushed, no response.
Copying the code pattern is really not a good method, it will make each
project bloated, and at the same time, when the copied source project
encounters the same problem, the user will deal with such problems
repeatedly.
How to activate these issues, or, is there anyone in charge who analyzes
our improvements, can we directly submit PR for review and integration?
-------------------------------------------------------------
Hi all,
I created some issues regarding to the discussion above,
https://github.com/ververica/flink-cdc-connectors/issues/1140
https://github.com/ververica/flink-cdc-connectors/issues/1138
https://github.com/ververica/flink-cdc-connectors/issues/1135
Zirui Sting <zi...@apache.org> 于2022年7月25日周一 11:14写道：
> Hi Jark,
>
> Thanks for your time and all
>
> Best,
> PENG
>
> On Mon, Jul 25, 2022 at 11:12 AM Jark Wu <im...@gmail.com> wrote:
>
> > Thanks for opening the issues!
> >
> > We will take a look and respond there.
> > Looking forward to cooperating with the InLong community!
> >
> > Best,
> > Jark
> >
> > On Mon, 25 Jul 2022 at 11:05, Charles Zhang <do...@apache.org>
> > wrote:
> >
> >> Hi, I am forwarding this email to make sure everyone gets the message.
> We
> >> can have a discussion and resolution of these issues directly.
> >>
> >> Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:53写道：
> >>
> >> > Hi all,
> >> >
> >> > I created some issues regarding to the discussion above,
> >> >
> >> > https://github.com/ververica/flink-cdc-connectors/issues/1140
> >> > https://github.com/ververica/flink-cdc-connectors/issues/1138
> >> > https://github.com/ververica/flink-cdc-connectors/issues/1135
> >> >
> >> > Maybe we can discuss them under these specific issues, thanks a lot !
> >> >
> >> > Yours,
> >> > PENG
> >> >
> >> >
> >> > On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com>
> wrote:
> >> >
> >> >> Thanks tison for starting the discussion here.
> >> >>
> >> >> Both Apache Inlong and Flink CDC Connectors are under Apache License
> >> 2.0
> >> >> protocol, It’s legal even someone copy/distribute all code only if
> >> he/she
> >> >> keeps the license files.
> >> >> But from the open source project’s long term maintaining perspective,
> >> >> personally I like upstream first way instead of copy/distribute
> way. I
> >> >> really hope Flink CDC project can help Apache
> >> >> InLong more and we’re welcome to open issue to statisfy the user
> >> >> requirements from inLong community.
> >> >>
> >> >>
> >> >> Best,
> >> >> Leonard Xu
> >> >>
> >> >> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> >> >> >
> >> >> > Hi, tison and Jark,
> >> >> > Thanks for your advice.
> >> >> > Different from Flink CDC connectors, InLong's changes to MySQL
> >> CDC
> >> >> connector mainly involve entire database migration and support for
> rich
> >> >> metadata fields (like canal), which are very important for Apache
> >> InLong to
> >> >> provide external services as data integration. For example, for the
> >> entire
> >> >> database migration, if one schema is used for one table, it will be a
> >> waste
> >> >> of resources and difficult to maintain, and it will also put pressure
> >> on
> >> >> the upstream database.We also particularly hope to work with Flink
> CDC
> >> to
> >> >> see how to support the entire database migration and support richer
> >> >> metadata fields, so that we can directly rely on it. We don't know if
> >> Flink
> >> >> CDC has plans and technical solutions for these two scenarios. We can
> >> talk
> >> >> about it together or contribute directly.
> >> >> > In addition, in order to speed up discussion and follow-up
> >> >> cooperation, I have created a WeChat group [1], and interested
> >> contributors
> >> >> can join the direct discussion.
> >> >> > Thanks again.
> >> >> > [1]
> >> >> > <image.png>
> >> >> >
> >> >> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> >> >> 22:45写道：
> >> >> > Thanks for starting this discussion, tison!
> >> >> >
> >> >> > It's great to see InLong is using flink-cdc-connectors to extract
> >> data
> >> >> from
> >> >> > various databases.
> >> >> > I think tison raised a good point about contributing back instead
> of
> >> >> > forking. I believe this can
> >> >> > help the InLong community in the long term by leveraging
> improvements
> >> >> and
> >> >> > bug fixes from
> >> >> > upstream. Flink CDC can also benefit from more contributions from
> the
> >> >> > InLong community.
> >> >> > I believe it's the only way that we can improve collaboration
> between
> >> >> > different OSS projects.
> >> >> >
> >> >> > In a word, I highly appreciate learning the missing part of Flink
> CDC
> >> >> that
> >> >> > InLong needs.
> >> >> > We can discuss the missing features here or in the issues.
> >> >> >
> >> >> > Cheers,
> >> >> > Jark
> >> >> >
> >> >> >
> >> >> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> >> >> wander4096@gmail.com>> wrote:
> >> >> >
> >> >> > > Hi,
> >> >> > >
> >> >> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied
> >> from
> >> >> > > Flink CDC mysql connector[2].
> >> >> > >
> >> >> > > With a short discussion with Charles I learned that InLong
> >> developers
> >> >> > > added metadata supports and migration of a whole database.
> >> >> > >
> >> >> > > It could be better to see if InLong community can contribute such
> >> >> supports
> >> >> > > back to Flink CDC upstream, and avoid maintaining a whole fork as
> >> >> well as
> >> >> > > periodically catch up upstream updates.
> >> >> > >
> >> >> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> >> >> > >
> >> >> > > Perhaps we can start with creating an issue on Flink CDC repo to
> >> >> describe
> >> >> > > the case, contribute back the improvements, and switch dependency
> >> to
> >> >> the
> >> >> > > upstream.
> >> >> > >
> >> >> > > Best,
> >> >> > > tison.
> >> >> > >
> >> >> > > [1]
> >> >> > >
> >> >>
> >>
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >> >> <
> >> >>
> >>
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >> >> >
> >> >> > > [2]
> >> >> > >
> >> >>
> >>
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >> >> <
> >> >>
> >>
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >> >> >
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Best wishes,
> >> >> > Charles Zhang
> >> >>
> >> >>
> >>
> >> --
> >> Best wishes,
> >> Charles Zhang
> >>
> >
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Zirui Sting <zi...@apache.org>.
Hi Jark,
Thanks for your time and all
Best,
PENG
On Mon, Jul 25, 2022 at 11:12 AM Jark Wu <im...@gmail.com> wrote:
> Thanks for opening the issues!
>
> We will take a look and respond there.
> Looking forward to cooperating with the InLong community!
>
> Best,
> Jark
>
> On Mon, 25 Jul 2022 at 11:05, Charles Zhang <do...@apache.org>
> wrote:
>
>> Hi, I am forwarding this email to make sure everyone gets the message. We
>> can have a discussion and resolution of these issues directly.
>>
>> Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:53写道：
>>
>> > Hi all,
>> >
>> > I created some issues regarding to the discussion above,
>> >
>> > https://github.com/ververica/flink-cdc-connectors/issues/1140
>> > https://github.com/ververica/flink-cdc-connectors/issues/1138
>> > https://github.com/ververica/flink-cdc-connectors/issues/1135
>> >
>> > Maybe we can discuss them under these specific issues, thanks a lot !
>> >
>> > Yours,
>> > PENG
>> >
>> >
>> > On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> >> Thanks tison for starting the discussion here.
>> >>
>> >> Both Apache Inlong and Flink CDC Connectors are under Apache License
>> 2.0
>> >> protocol, It’s legal even someone copy/distribute all code only if
>> he/she
>> >> keeps the license files.
>> >> But from the open source project’s long term maintaining perspective,
>> >> personally I like upstream first way instead of copy/distribute way. I
>> >> really hope Flink CDC project can help Apache
>> >> InLong more and we’re welcome to open issue to statisfy the user
>> >> requirements from inLong community.
>> >>
>> >>
>> >> Best,
>> >> Leonard Xu
>> >>
>> >> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
>> >> >
>> >> > Hi, tison and Jark,
>> >> > Thanks for your advice.
>> >> > Different from Flink CDC connectors, InLong's changes to MySQL
>> CDC
>> >> connector mainly involve entire database migration and support for rich
>> >> metadata fields (like canal), which are very important for Apache
>> InLong to
>> >> provide external services as data integration. For example, for the
>> entire
>> >> database migration, if one schema is used for one table, it will be a
>> waste
>> >> of resources and difficult to maintain, and it will also put pressure
>> on
>> >> the upstream database.We also particularly hope to work with Flink CDC
>> to
>> >> see how to support the entire database migration and support richer
>> >> metadata fields, so that we can directly rely on it. We don't know if
>> Flink
>> >> CDC has plans and technical solutions for these two scenarios. We can
>> talk
>> >> about it together or contribute directly.
>> >> > In addition, in order to speed up discussion and follow-up
>> >> cooperation, I have created a WeChat group [1], and interested
>> contributors
>> >> can join the direct discussion.
>> >> > Thanks again.
>> >> > [1]
>> >> > <image.png>
>> >> >
>> >> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
>> >> 22:45写道：
>> >> > Thanks for starting this discussion, tison!
>> >> >
>> >> > It's great to see InLong is using flink-cdc-connectors to extract
>> data
>> >> from
>> >> > various databases.
>> >> > I think tison raised a good point about contributing back instead of
>> >> > forking. I believe this can
>> >> > help the InLong community in the long term by leveraging improvements
>> >> and
>> >> > bug fixes from
>> >> > upstream. Flink CDC can also benefit from more contributions from the
>> >> > InLong community.
>> >> > I believe it's the only way that we can improve collaboration between
>> >> > different OSS projects.
>> >> >
>> >> > In a word, I highly appreciate learning the missing part of Flink CDC
>> >> that
>> >> > InLong needs.
>> >> > We can discuss the missing features here or in the issues.
>> >> >
>> >> > Cheers,
>> >> > Jark
>> >> >
>> >> >
>> >> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
>> >> wander4096@gmail.com>> wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied
>> from
>> >> > > Flink CDC mysql connector[2].
>> >> > >
>> >> > > With a short discussion with Charles I learned that InLong
>> developers
>> >> > > added metadata supports and migration of a whole database.
>> >> > >
>> >> > > It could be better to see if InLong community can contribute such
>> >> supports
>> >> > > back to Flink CDC upstream, and avoid maintaining a whole fork as
>> >> well as
>> >> > > periodically catch up upstream updates.
>> >> > >
>> >> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
>> >> > >
>> >> > > Perhaps we can start with creating an issue on Flink CDC repo to
>> >> describe
>> >> > > the case, contribute back the improvements, and switch dependency
>> to
>> >> the
>> >> > > upstream.
>> >> > >
>> >> > > Best,
>> >> > > tison.
>> >> > >
>> >> > > [1]
>> >> > >
>> >>
>> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
>> >> <
>> >>
>> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
>> >> >
>> >> > > [2]
>> >> > >
>> >>
>> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>> >> <
>> >>
>> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>> >> >
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > Best wishes,
>> >> > Charles Zhang
>> >>
>> >>
>>
>> --
>> Best wishes,
>> Charles Zhang
>>
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Jark Wu <im...@gmail.com>.
Thanks for opening the issues!
We will take a look and respond there.
Looking forward to cooperating with the InLong community!
Best,
Jark
On Mon, 25 Jul 2022 at 11:05, Charles Zhang <do...@apache.org> wrote:
> Hi, I am forwarding this email to make sure everyone gets the message. We
> can have a discussion and resolution of these issues directly.
>
> Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:53写道：
>
> > Hi all,
> >
> > I created some issues regarding to the discussion above,
> >
> > https://github.com/ververica/flink-cdc-connectors/issues/1140
> > https://github.com/ververica/flink-cdc-connectors/issues/1138
> > https://github.com/ververica/flink-cdc-connectors/issues/1135
> >
> > Maybe we can discuss them under these specific issues, thanks a lot !
> >
> > Yours,
> > PENG
> >
> >
> > On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
> >
> >> Thanks tison for starting the discussion here.
> >>
> >> Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0
> >> protocol, It’s legal even someone copy/distribute all code only if
> he/she
> >> keeps the license files.
> >> But from the open source project’s long term maintaining perspective,
> >> personally I like upstream first way instead of copy/distribute way. I
> >> really hope Flink CDC project can help Apache
> >> InLong more and we’re welcome to open issue to statisfy the user
> >> requirements from inLong community.
> >>
> >>
> >> Best,
> >> Leonard Xu
> >>
> >> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> >> >
> >> > Hi, tison and Jark,
> >> > Thanks for your advice.
> >> > Different from Flink CDC connectors, InLong's changes to MySQL CDC
> >> connector mainly involve entire database migration and support for rich
> >> metadata fields (like canal), which are very important for Apache
> InLong to
> >> provide external services as data integration. For example, for the
> entire
> >> database migration, if one schema is used for one table, it will be a
> waste
> >> of resources and difficult to maintain, and it will also put pressure on
> >> the upstream database.We also particularly hope to work with Flink CDC
> to
> >> see how to support the entire database migration and support richer
> >> metadata fields, so that we can directly rely on it. We don't know if
> Flink
> >> CDC has plans and technical solutions for these two scenarios. We can
> talk
> >> about it together or contribute directly.
> >> > In addition, in order to speed up discussion and follow-up
> >> cooperation, I have created a WeChat group [1], and interested
> contributors
> >> can join the direct discussion.
> >> > Thanks again.
> >> > [1]
> >> > <image.png>
> >> >
> >> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> >> 22:45写道：
> >> > Thanks for starting this discussion, tison!
> >> >
> >> > It's great to see InLong is using flink-cdc-connectors to extract data
> >> from
> >> > various databases.
> >> > I think tison raised a good point about contributing back instead of
> >> > forking. I believe this can
> >> > help the InLong community in the long term by leveraging improvements
> >> and
> >> > bug fixes from
> >> > upstream. Flink CDC can also benefit from more contributions from the
> >> > InLong community.
> >> > I believe it's the only way that we can improve collaboration between
> >> > different OSS projects.
> >> >
> >> > In a word, I highly appreciate learning the missing part of Flink CDC
> >> that
> >> > InLong needs.
> >> > We can discuss the missing features here or in the issues.
> >> >
> >> > Cheers,
> >> > Jark
> >> >
> >> >
> >> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> >> wander4096@gmail.com>> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied
> from
> >> > > Flink CDC mysql connector[2].
> >> > >
> >> > > With a short discussion with Charles I learned that InLong
> developers
> >> > > added metadata supports and migration of a whole database.
> >> > >
> >> > > It could be better to see if InLong community can contribute such
> >> supports
> >> > > back to Flink CDC upstream, and avoid maintaining a whole fork as
> >> well as
> >> > > periodically catch up upstream updates.
> >> > >
> >> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> >> > >
> >> > > Perhaps we can start with creating an issue on Flink CDC repo to
> >> describe
> >> > > the case, contribute back the improvements, and switch dependency to
> >> the
> >> > > upstream.
> >> > >
> >> > > Best,
> >> > > tison.
> >> > >
> >> > > [1]
> >> > >
> >>
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >> <
> >>
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >> >
> >> > > [2]
> >> > >
> >>
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >> <
> >>
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >> >
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Best wishes,
> >> > Charles Zhang
> >>
> >>
>
> --
> Best wishes,
> Charles Zhang
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Charles Zhang <do...@apache.org>.
Hi, I am forwarding this email to make sure everyone gets the message. We
can have a discussion and resolution of these issues directly.
Zirui Sting <zi...@apache.org> 于2022年7月25日周一 10:53写道：
> Hi all,
>
> I created some issues regarding to the discussion above,
>
> https://github.com/ververica/flink-cdc-connectors/issues/1140
> https://github.com/ververica/flink-cdc-connectors/issues/1138
> https://github.com/ververica/flink-cdc-connectors/issues/1135
>
> Maybe we can discuss them under these specific issues, thanks a lot !
>
> Yours,
> PENG
>
>
> On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
>
>> Thanks tison for starting the discussion here.
>>
>> Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0
>> protocol, It’s legal even someone copy/distribute all code only if he/she
>> keeps the license files.
>> But from the open source project’s long term maintaining perspective,
>> personally I like upstream first way instead of copy/distribute way. I
>> really hope Flink CDC project can help Apache
>> InLong more and we’re welcome to open issue to statisfy the user
>> requirements from inLong community.
>>
>>
>> Best,
>> Leonard Xu
>>
>> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
>> >
>> > Hi, tison and Jark,
>> > Thanks for your advice.
>> > Different from Flink CDC connectors, InLong's changes to MySQL CDC
>> connector mainly involve entire database migration and support for rich
>> metadata fields (like canal), which are very important for Apache InLong to
>> provide external services as data integration. For example, for the entire
>> database migration, if one schema is used for one table, it will be a waste
>> of resources and difficult to maintain, and it will also put pressure on
>> the upstream database.We also particularly hope to work with Flink CDC to
>> see how to support the entire database migration and support richer
>> metadata fields, so that we can directly rely on it. We don't know if Flink
>> CDC has plans and technical solutions for these two scenarios. We can talk
>> about it together or contribute directly.
>> > In addition, in order to speed up discussion and follow-up
>> cooperation, I have created a WeChat group [1], and interested contributors
>> can join the direct discussion.
>> > Thanks again.
>> > [1]
>> > <image.png>
>> >
>> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
>> 22:45写道：
>> > Thanks for starting this discussion, tison!
>> >
>> > It's great to see InLong is using flink-cdc-connectors to extract data
>> from
>> > various databases.
>> > I think tison raised a good point about contributing back instead of
>> > forking. I believe this can
>> > help the InLong community in the long term by leveraging improvements
>> and
>> > bug fixes from
>> > upstream. Flink CDC can also benefit from more contributions from the
>> > InLong community.
>> > I believe it's the only way that we can improve collaboration between
>> > different OSS projects.
>> >
>> > In a word, I highly appreciate learning the missing part of Flink CDC
>> that
>> > InLong needs.
>> > We can discuss the missing features here or in the issues.
>> >
>> > Cheers,
>> > Jark
>> >
>> >
>> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
>> wander4096@gmail.com>> wrote:
>> >
>> > > Hi,
>> > >
>> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
>> > > Flink CDC mysql connector[2].
>> > >
>> > > With a short discussion with Charles I learned that InLong developers
>> > > added metadata supports and migration of a whole database.
>> > >
>> > > It could be better to see if InLong community can contribute such
>> supports
>> > > back to Flink CDC upstream, and avoid maintaining a whole fork as
>> well as
>> > > periodically catch up upstream updates.
>> > >
>> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
>> > >
>> > > Perhaps we can start with creating an issue on Flink CDC repo to
>> describe
>> > > the case, contribute back the improvements, and switch dependency to
>> the
>> > > upstream.
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > > [1]
>> > >
>> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
>> <
>> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
>> >
>> > > [2]
>> > >
>> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>> <
>> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>> >
>> > >
>> > >
>> >
>> >
>> > --
>> > Best wishes,
>> > Charles Zhang
>>
>>
--
Best wishes,
Charles Zhang
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Zirui Sting <zi...@apache.org>.
Hi all,
I created some issues regarding to the discussion above,
https://github.com/ververica/flink-cdc-connectors/issues/1140
https://github.com/ververica/flink-cdc-connectors/issues/1138
https://github.com/ververica/flink-cdc-connectors/issues/1135
Maybe we can discuss them under these specific issues, thanks a lot !
Yours,
PENG
On Mon, Jul 25, 2022 at 10:19 AM Leonard Xu <xb...@gmail.com> wrote:
> Thanks tison for starting the discussion here.
>
> Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0
> protocol, It’s legal even someone copy/distribute all code only if he/she
> keeps the license files.
> But from the open source project’s long term maintaining perspective,
> personally I like upstream first way instead of copy/distribute way. I
> really hope Flink CDC project can help Apache
> InLong more and we’re welcome to open issue to statisfy the user
> requirements from inLong community.
>
>
> Best,
> Leonard Xu
>
> > 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
> >
> > Hi, tison and Jark,
> > Thanks for your advice.
> > Different from Flink CDC connectors, InLong's changes to MySQL CDC
> connector mainly involve entire database migration and support for rich
> metadata fields (like canal), which are very important for Apache InLong to
> provide external services as data integration. For example, for the entire
> database migration, if one schema is used for one table, it will be a waste
> of resources and difficult to maintain, and it will also put pressure on
> the upstream database.We also particularly hope to work with Flink CDC to
> see how to support the entire database migration and support richer
> metadata fields, so that we can directly rely on it. We don't know if Flink
> CDC has plans and technical solutions for these two scenarios. We can talk
> about it together or contribute directly.
> > In addition, in order to speed up discussion and follow-up
> cooperation, I have created a WeChat group [1], and interested contributors
> can join the direct discussion.
> > Thanks again.
> > [1]
> > <image.png>
> >
> > Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六
> 22:45写道：
> > Thanks for starting this discussion, tison!
> >
> > It's great to see InLong is using flink-cdc-connectors to extract data
> from
> > various databases.
> > I think tison raised a good point about contributing back instead of
> > forking. I believe this can
> > help the InLong community in the long term by leveraging improvements and
> > bug fixes from
> > upstream. Flink CDC can also benefit from more contributions from the
> > InLong community.
> > I believe it's the only way that we can improve collaboration between
> > different OSS projects.
> >
> > In a word, I highly appreciate learning the missing part of Flink CDC
> that
> > InLong needs.
> > We can discuss the missing features here or in the issues.
> >
> > Cheers,
> > Jark
> >
> >
> > On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <mailto:
> wander4096@gmail.com>> wrote:
> >
> > > Hi,
> > >
> > > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> > > Flink CDC mysql connector[2].
> > >
> > > With a short discussion with Charles I learned that InLong developers
> > > added metadata supports and migration of a whole database.
> > >
> > > It could be better to see if InLong community can contribute such
> supports
> > > back to Flink CDC upstream, and avoid maintaining a whole fork as well
> as
> > > periodically catch up upstream updates.
> > >
> > > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> > >
> > > Perhaps we can start with creating an issue on Flink CDC repo to
> describe
> > > the case, contribute back the improvements, and switch dependency to
> the
> > > upstream.
> > >
> > > Best,
> > > tison.
> > >
> > > [1]
> > >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> <
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> >
> > > [2]
> > >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> <
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >
> > >
> > >
> >
> >
> > --
> > Best wishes,
> > Charles Zhang
>
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Leonard Xu <xb...@gmail.com>.
Thanks tison for starting the discussion here.
Both Apache Inlong and Flink CDC Connectors are under Apache License 2.0 protocol, It’s legal even someone copy/distribute all code only if he/she keeps the license files.
But from the open source project’s long term maintaining perspective, personally I like upstream first way instead of copy/distribute way. I really hope Flink CDC project can help Apache
InLong more and we’re welcome to open issue to statisfy the user requirements from inLong community.
Best,
Leonard Xu
> 2022年7月24日 下午4:56，Charles Zhang <do...@apache.org> 写道：
>
> Hi, tison and Jark,
> Thanks for your advice.
> Different from Flink CDC connectors, InLong's changes to MySQL CDC connector mainly involve entire database migration and support for rich metadata fields (like canal), which are very important for Apache InLong to provide external services as data integration. For example, for the entire database migration, if one schema is used for one table, it will be a waste of resources and difficult to maintain, and it will also put pressure on the upstream database.We also particularly hope to work with Flink CDC to see how to support the entire database migration and support richer metadata fields, so that we can directly rely on it. We don't know if Flink CDC has plans and technical solutions for these two scenarios. We can talk about it together or contribute directly.
> In addition, in order to speed up discussion and follow-up cooperation, I have created a WeChat group [1], and interested contributors can join the direct discussion.
> Thanks again.
> [1]
> <image.png>
>
> Jark Wu <imjark@gmail.com <ma...@gmail.com>> 于2022年7月23日周六 22:45写道：
> Thanks for starting this discussion, tison!
>
> It's great to see InLong is using flink-cdc-connectors to extract data from
> various databases.
> I think tison raised a good point about contributing back instead of
> forking. I believe this can
> help the InLong community in the long term by leveraging improvements and
> bug fixes from
> upstream. Flink CDC can also benefit from more contributions from the
> InLong community.
> I believe it's the only way that we can improve collaboration between
> different OSS projects.
>
> In a word, I highly appreciate learning the missing part of Flink CDC that
> InLong needs.
> We can discuss the missing features here or in the issues.
>
> Cheers,
> Jark
>
>
> On Fri, 22 Jul 2022 at 23:56, tison <wander4096@gmail.com <ma...@gmail.com>> wrote:
>
> > Hi,
> >
> > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> > Flink CDC mysql connector[2].
> >
> > With a short discussion with Charles I learned that InLong developers
> > added metadata supports and migration of a whole database.
> >
> > It could be better to see if InLong community can contribute such supports
> > back to Flink CDC upstream, and avoid maintaining a whole fork as well as
> > periodically catch up upstream updates.
> >
> > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> >
> > Perhaps we can start with creating an issue on Flink CDC repo to describe
> > the case, contribute back the improvements, and switch dependency to the
> > upstream.
> >
> > Best,
> > tison.
> >
> > [1]
> > https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc <https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc>
> > [2]
> > https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc <https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc>
> >
> >
>
>
> --
> Best wishes,
> Charles Zhang
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by tison <wa...@gmail.com>.
Hi,
I second to Jark's suggestion to continue the discussion and task on
mailing list or an issue.
Since the diff is clear from InLong side, a well described issue should be
a good start point. May Charles or the author of these changes create one?
Apart from that the QR code photo is broken on mailing list, I tend not to
use such private channel. Basically, open-source collaboration happens in
public and everyone is potential contributor. It's not we call several
associated employees to finish jobs with a hard estimate.
Best,
tison.
Charles Zhang <do...@apache.org> 于2022年7月24日周日 16:56写道：
> Hi, tison and Jark,
> Thanks for your advice.
> Different from Flink CDC connectors, InLong's changes to MySQL CDC
> connector mainly involve entire database migration and support for rich
> metadata fields (like canal), which are very important for Apache InLong to
> provide external services as data integration. For example, for the entire
> database migration, if one schema is used for one table, it will be a waste
> of resources and difficult to maintain, and it will also put pressure on
> the upstream database.We also particularly hope to work with Flink CDC to
> see how to support the entire database migration and support richer
> metadata fields, so that we can directly rely on it. We don't know if Flink
> CDC has plans and technical solutions for these two scenarios. We can talk
> about it together or contribute directly.
> In addition, in order to speed up discussion and follow-up
> cooperation, I have created a WeChat group [1], and interested contributors
> can join the direct discussion.
> Thanks again.
> [1]
> [image: image.png]
>
> Jark Wu <im...@gmail.com> 于2022年7月23日周六 22:45写道：
>
>> Thanks for starting this discussion, tison!
>>
>> It's great to see InLong is using flink-cdc-connectors to extract data
>> from
>> various databases.
>> I think tison raised a good point about contributing back instead of
>> forking. I believe this can
>> help the InLong community in the long term by leveraging improvements and
>> bug fixes from
>> upstream. Flink CDC can also benefit from more contributions from the
>> InLong community.
>> I believe it's the only way that we can improve collaboration between
>> different OSS projects.
>>
>> In a word, I highly appreciate learning the missing part of Flink CDC that
>> InLong needs.
>> We can discuss the missing features here or in the issues.
>>
>> Cheers,
>> Jark
>>
>>
>> On Fri, 22 Jul 2022 at 23:56, tison <wa...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
>> > Flink CDC mysql connector[2].
>> >
>> > With a short discussion with Charles I learned that InLong developers
>> > added metadata supports and migration of a whole database.
>> >
>> > It could be better to see if InLong community can contribute such
>> supports
>> > back to Flink CDC upstream, and avoid maintaining a whole fork as well
>> as
>> > periodically catch up upstream updates.
>> >
>> > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
>> >
>> > Perhaps we can start with creating an issue on Flink CDC repo to
>> describe
>> > the case, contribute back the improvements, and switch dependency to the
>> > upstream.
>> >
>> > Best,
>> > tison.
>> >
>> > [1]
>> >
>> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
>> > [2]
>> >
>> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>> >
>> >
>>
>
>
> --
> Best wishes,
> Charles Zhang
>
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Charles Zhang <do...@apache.org>.
Hi, tison and Jark,
Thanks for your advice.
Different from Flink CDC connectors, InLong's changes to MySQL CDC
connector mainly involve entire database migration and support for rich
metadata fields (like canal), which are very important for Apache InLong to
provide external services as data integration. For example, for the entire
database migration, if one schema is used for one table, it will be a waste
of resources and difficult to maintain, and it will also put pressure on
the upstream database.We also particularly hope to work with Flink CDC to
see how to support the entire database migration and support richer
metadata fields, so that we can directly rely on it. We don't know if Flink
CDC has plans and technical solutions for these two scenarios. We can talk
about it together or contribute directly.
In addition, in order to speed up discussion and follow-up cooperation,
I have created a WeChat group [1], and interested contributors can join the
direct discussion.
Thanks again.
[1]
[image: image.png]
Jark Wu <im...@gmail.com> 于2022年7月23日周六 22:45写道：
> Thanks for starting this discussion, tison!
>
> It's great to see InLong is using flink-cdc-connectors to extract data from
> various databases.
> I think tison raised a good point about contributing back instead of
> forking. I believe this can
> help the InLong community in the long term by leveraging improvements and
> bug fixes from
> upstream. Flink CDC can also benefit from more contributions from the
> InLong community.
> I believe it's the only way that we can improve collaboration between
> different OSS projects.
>
> In a word, I highly appreciate learning the missing part of Flink CDC that
> InLong needs.
> We can discuss the missing features here or in the issues.
>
> Cheers,
> Jark
>
>
> On Fri, 22 Jul 2022 at 23:56, tison <wa...@gmail.com> wrote:
>
> > Hi,
> >
> > I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> > Flink CDC mysql connector[2].
> >
> > With a short discussion with Charles I learned that InLong developers
> > added metadata supports and migration of a whole database.
> >
> > It could be better to see if InLong community can contribute such
> supports
> > back to Flink CDC upstream, and avoid maintaining a whole fork as well as
> > periodically catch up upstream updates.
> >
> > Maintainers of Flink CDC projects are in cc (Leonard & Jark).
> >
> > Perhaps we can start with creating an issue on Flink CDC repo to describe
> > the case, contribute back the improvements, and switch dependency to the
> > upstream.
> >
> > Best,
> > tison.
> >
> > [1]
> >
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> > [2]
> >
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
> >
> >
>
--
Best wishes,
Charles Zhang
Re: [DISCUSS] Contribute back modifications of MySQL CDC connector
Posted by Jark Wu <im...@gmail.com>.
Thanks for starting this discussion, tison!
It's great to see InLong is using flink-cdc-connectors to extract data from
various databases.
I think tison raised a good point about contributing back instead of
forking. I believe this can
help the InLong community in the long term by leveraging improvements and
bug fixes from
upstream. Flink CDC can also benefit from more contributions from the
InLong community.
I believe it's the only way that we can improve collaboration between
different OSS projects.
In a word, I highly appreciate learning the missing part of Flink CDC that
InLong needs.
We can discuss the missing features here or in the issues.
Cheers,
Jark
On Fri, 22 Jul 2022 at 23:56, tison <wa...@gmail.com> wrote:
> Hi,
>
> I notice that inlong-sort mysql-cdc connector[1] is almost copied from
> Flink CDC mysql connector[2].
>
> With a short discussion with Charles I learned that InLong developers
> added metadata supports and migration of a whole database.
>
> It could be better to see if InLong community can contribute such supports
> back to Flink CDC upstream, and avoid maintaining a whole fork as well as
> periodically catch up upstream updates.
>
> Maintainers of Flink CDC projects are in cc (Leonard & Jark).
>
> Perhaps we can start with creating an issue on Flink CDC repo to describe
> the case, contribute back the improvements, and switch dependency to the
> upstream.
>
> Best,
> tison.
>
> [1]
> https://github.com/apache/inlong/tree/master/inlong-sort/sort-connectors/mysql-cdc
> [2]
> https://github.com/ververica/flink-cdc-connectors/tree/master/flink-connector-mysql-cdc
>
>