You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Leonard Xu <xb...@gmail.com> on 2023/12/07 03:24:59 UTC

[PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Dear Flink devs,

As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.

CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.


Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].


I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.

Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 


While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.


Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.


Really looking forward to hear what you think! 


Best,
Leonard (on behalf of the Flink CDC Connectors project maintainers)

[1] https://github.com/ververica/flink-cdc-connectors
[2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
[3] https://debezium.io
[4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Nicholas Jiang <ni...@apache.org>.
Awesome! Big +1.

Regards,
Nicholas Jiang


On 2023/12/07 03:24:59 Leonard Xu wrote:
> Dear Flink devs,
> 
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> 
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> 
> 
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> 
> 
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> 
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 
> 
> 
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> 
> 
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> 
> 
> Really looking forward to hear what you think! 
> 
> 
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> 
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Zirui Peng <zi...@apache.org>.
+1 for me,

As a contributor in Flink CDC, I've witness so much excellent work that Flink CDC community have done for the latest years. Also Apache InLong benefits a lot on Flink CDC for its ability to support database changes. Much appreciation for the Flink CDC community and its maintainers.

It's a great pleasure to support this contribution.

Best,
Zirui.

On 2023/12/08 07:10:48 Yu Li wrote:
> +1
> 
> Thanks Leonard for the proposal, and all Flink CDC contributors for the
> existing great works. I believe these two projects are a perfect fit and
> the donation will benifit both.
> 
> Best Regards,
> Yu
> 
> 
> On Fri, Dec 8, 2023 at 2:49 PM bai_wentao1 <ba...@163.com> wrote:
> 
> > +1 for this exciting work
> >
> >
> > Best regards,
> > WT
> >
> >
> > | |
> > bai_wentao1
> > |
> > |
> > bai_wentao1@163.com
> > |
> > ---- Replied Message ----
> > | From | ConradJam<ja...@gmail.com> |
> > | Date | 12/8/2023 14:26 |
> > | To | <de...@flink.apache.org> |
> > | Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to
> > Apache Flink |
> > +1 Best Idea . Thanks to all the contributors of Flink CDC for the great
> > work.
> >
> > yuxia <lu...@alumni.sjtu.edu.cn> 于2023年12月7日周四 20:46写道：
> >
> > +1 for this. Thanks all the contributors of Flink CDC for the great work.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Martijn Visser" <ma...@apache.org>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36
> > 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
> >
> > Hi Leonard,
> >
> > +1 for this. I think the CDC connectors are a great result and example
> > of the Flink CDC community. Kudos to you all.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com>
> > wrote:
> >
> > Hi Leonard,
> >
> > Thank you for the excellent work you and the team working on the CDC
> > connectors project have been doing so far. I am +1 of having them under
> > Flink's umbrella.
> >
> > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
> > wrote:
> >
> > Big +1, thanks this will be a very useful addition to Flink.
> >
> > Best
> >
> > Etienne
> >
> > Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > +1 for contributing CDC Connectors  to Apache Flink.
> >
> > Best,
> > Hang
> >
> > Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
> >
> > Cool, +1 for contributing CDC Connectors to Apache Flink.
> >
> > Best,
> > Yuxin
> >
> >
> > Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
> >
> > Awesome! +1
> >
> > Best regards,
> > Jing
> >
> > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<snuyanzin@gmail.com
> >
> > wrote:
> >
> > thanks for working on this and driving it
> >
> > +1
> >
> > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> > wrote:
> >
> > This is incredibly exciting news, a big +1 for this.
> >
> > Thank you for the fantastic work on Flink CDC. We have created
> > thousands
> > of
> > real-time integration jobs using Flink CDC connectors.
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> > gongzhongqiang@apache.org
> > wrote:
> >
> > It's very exciting to hear the news.
> > +1 for adding CDC Connectors  to Apache Flink !
> >
> >
> > Best,
> > Zhongqiang
> >
> > Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
> >
> > Dear Flink devs,
> >
> >
> > As you may have heard, we at Alibaba (Ververica) are planning
> > to
> > donate
> > CDC Connectors for the Apache Flink project
> > *[1]* to the Apache Flink community.
> >
> >
> >
> > CDC Connectors for Apache Flink comprise a collection of source
> > connectors designed specifically for Apache Flink. These
> > connectors
> > *[2]*
> > enable the ingestion of changes from various databases using
> > Change
> > Data Capture (CDC), most of these CDC connectors are powered by
> > Debezium
> > *[3]*
> > . They support both the DataStream API and the Table/SQL API,
> > facilitating the reading of database snapshots and continuous
> > reading
> > of
> > transaction logs with exactly-once processing, even in the
> > event of
> > failures.
> >
> >
> > Additionally, in the latest version 3.0, we have introduced
> > many
> > long-awaited features. Starting from CDC version 3.0, we've
> > built a
> > Streaming ELT Framework available for streaming data
> > integration.
> > This
> > framework allows users to write their data synchronization logic
> > in a
> > simple YAML file, which will automatically be translated into a
> > Flink
> > DataStreaming job. It emphasizes optimizing the task submission
> > process
> > and
> > offers advanced functionalities such as whole database
> > synchronization,
> > merging sharded tables, and schema evolution
> > *[4]*.
> >
> >
> >
> >
> > I believe this initiative is a perfect match for both sides.
> > For
> > the
> > Flink community, it presents an opportunity to enhance Flink's
> > competitive
> > advantage in streaming data integration, promoting the healthy
> > growth
> > and
> > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > project,
> > becoming a sub-project of Apache Flink means being part of a
> > neutral
> > open-source community, which can attract a more diverse pool of
> > contributors.
> >
> > Please note that the aforementioned points represent only some
> > of
> > our
> > motivations and vision for this donation. Specific future
> > operations
> > need
> > to be further discussed in this thread. For example, the
> > sub-project
> > name
> > after the donation; we hope to name it Flink-CDC
> > aiming to streaming data intergration through Apache Flink,
> > following the naming convention of Flink-ML; And this project
> > is
> > managed
> > by a total of 8 maintainers, including 3 Flink PMC members and 1
> > Flink
> > Committer. The remaining 4 maintainers are also highly active
> > contributors
> > to the Flink community, donating this project to the Flink
> > community
> > implies that their permissions might be reduced. Therefore, we
> > may
> > need
> > to
> > bring up this topic for further discussion within the Flink PMC.
> > Additionally, we need to discuss how to migrate existing users
> > and
> > documents. We have a user group of nearly 10,000 people and a
> > multi-version
> > documentation site need to migrate. We also need to plan for the
> > migration
> > of CI/CD processes and other specifics.
> >
> >
> > While there are many intricate details that require
> > implementation,
> > we
> > are committed to progressing and finalizing this donation
> > process.
> >
> >
> > Despite being Flink’s most active ecological project (as
> > evaluated
> > by
> > GitHub metrics), it also boasts a significant user base.
> > However, I
> > believe
> > it's essential to commence discussions on future operations only
> > after
> > the
> > community reaches a consensus on whether they desire this
> > donation.
> >
> > Really looking forward to hear what you think!
> >
> >
> >
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project
> > maintainers)
> > [1]https://github.com/ververica/flink-cdc-connectors
> > [2]
> >
> >
> >
> >
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3]https://debezium.io
> > [4]
> >
> >
> >
> >
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
> > --
> > Best regards,
> > Sergey
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Yu Li <ca...@gmail.com>.
+1

Thanks Leonard for the proposal, and all Flink CDC contributors for the
existing great works. I believe these two projects are a perfect fit and
the donation will benifit both.

Best Regards,
Yu


On Fri, Dec 8, 2023 at 2:49 PM bai_wentao1 <ba...@163.com> wrote:

> +1 for this exciting work
>
>
> Best regards,
> WT
>
>
> | |
> bai_wentao1
> |
> |
> bai_wentao1@163.com
> |
> ---- Replied Message ----
> | From | ConradJam<ja...@gmail.com> |
> | Date | 12/8/2023 14:26 |
> | To | <de...@flink.apache.org> |
> | Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to
> Apache Flink |
> +1 Best Idea . Thanks to all the contributors of Flink CDC for the great
> work.
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2023年12月7日周四 20:46写道：
>
> +1 for this. Thanks all the contributors of Flink CDC for the great work.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Martijn Visser" <ma...@apache.org>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36
> 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
>
> Hi Leonard,
>
> +1 for this. I think the CDC connectors are a great result and example
> of the Flink CDC community. Kudos to you all.
>
> Best regards,
>
> Martijn
>
> On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
> Hi Leonard,
>
> Thank you for the excellent work you and the team working on the CDC
> connectors project have been doing so far. I am +1 of having them under
> Flink's umbrella.
>
> On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
> Big +1, thanks this will be a very useful addition to Flink.
>
> Best
>
> Etienne
>
> Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> +1 for contributing CDC Connectors  to Apache Flink.
>
> Best,
> Hang
>
> Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
>
> Cool, +1 for contributing CDC Connectors to Apache Flink.
>
> Best,
> Yuxin
>
>
> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
>
> Awesome! +1
>
> Best regards,
> Jing
>
> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<snuyanzin@gmail.com
>
> wrote:
>
> thanks for working on this and driving it
>
> +1
>
> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> wrote:
>
> This is incredibly exciting news, a big +1 for this.
>
> Thank you for the fantastic work on Flink CDC. We have created
> thousands
> of
> real-time integration jobs using Flink CDC connectors.
>
>
> Best,
> Feng
>
> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> gongzhongqiang@apache.org
> wrote:
>
> It's very exciting to hear the news.
> +1 for adding CDC Connectors  to Apache Flink !
>
>
> Best,
> Zhongqiang
>
> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
>
> Dear Flink devs,
>
>
> As you may have heard, we at Alibaba (Ververica) are planning
> to
> donate
> CDC Connectors for the Apache Flink project
> *[1]* to the Apache Flink community.
>
>
>
> CDC Connectors for Apache Flink comprise a collection of source
> connectors designed specifically for Apache Flink. These
> connectors
> *[2]*
> enable the ingestion of changes from various databases using
> Change
> Data Capture (CDC), most of these CDC connectors are powered by
> Debezium
> *[3]*
> . They support both the DataStream API and the Table/SQL API,
> facilitating the reading of database snapshots and continuous
> reading
> of
> transaction logs with exactly-once processing, even in the
> event of
> failures.
>
>
> Additionally, in the latest version 3.0, we have introduced
> many
> long-awaited features. Starting from CDC version 3.0, we've
> built a
> Streaming ELT Framework available for streaming data
> integration.
> This
> framework allows users to write their data synchronization logic
> in a
> simple YAML file, which will automatically be translated into a
> Flink
> DataStreaming job. It emphasizes optimizing the task submission
> process
> and
> offers advanced functionalities such as whole database
> synchronization,
> merging sharded tables, and schema evolution
> *[4]*.
>
>
>
>
> I believe this initiative is a perfect match for both sides.
> For
> the
> Flink community, it presents an opportunity to enhance Flink's
> competitive
> advantage in streaming data integration, promoting the healthy
> growth
> and
> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> project,
> becoming a sub-project of Apache Flink means being part of a
> neutral
> open-source community, which can attract a more diverse pool of
> contributors.
>
> Please note that the aforementioned points represent only some
> of
> our
> motivations and vision for this donation. Specific future
> operations
> need
> to be further discussed in this thread. For example, the
> sub-project
> name
> after the donation; we hope to name it Flink-CDC
> aiming to streaming data intergration through Apache Flink,
> following the naming convention of Flink-ML; And this project
> is
> managed
> by a total of 8 maintainers, including 3 Flink PMC members and 1
> Flink
> Committer. The remaining 4 maintainers are also highly active
> contributors
> to the Flink community, donating this project to the Flink
> community
> implies that their permissions might be reduced. Therefore, we
> may
> need
> to
> bring up this topic for further discussion within the Flink PMC.
> Additionally, we need to discuss how to migrate existing users
> and
> documents. We have a user group of nearly 10,000 people and a
> multi-version
> documentation site need to migrate. We also need to plan for the
> migration
> of CI/CD processes and other specifics.
>
>
> While there are many intricate details that require
> implementation,
> we
> are committed to progressing and finalizing this donation
> process.
>
>
> Despite being Flink’s most active ecological project (as
> evaluated
> by
> GitHub metrics), it also boasts a significant user base.
> However, I
> believe
> it's essential to commence discussions on future operations only
> after
> the
> community reaches a consensus on whether they desire this
> donation.
>
> Really looking forward to hear what you think!
>
>
>
> Best,
> Leonard (on behalf of the Flink CDC Connectors project
> maintainers)
> [1]https://github.com/ververica/flink-cdc-connectors
> [2]
>
>
>
>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3]https://debezium.io
> [4]
>
>
>
>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>
> --
> Best regards,
> Sergey
>
>
>
>
> --
> Best
>
> ConradJam
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by bai_wentao1 <ba...@163.com>.
+1 for this exciting work


Best regards,
WT


| |
bai_wentao1
|
|
bai_wentao1@163.com
|
---- Replied Message ----
| From | ConradJam<ja...@gmail.com> |
| Date | 12/8/2023 14:26 |
| To | <de...@flink.apache.org> |
| Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink |
+1 Best Idea . Thanks to all the contributors of Flink CDC for the great
work.

yuxia <lu...@alumni.sjtu.edu.cn> 于2023年12月7日周四 20:46写道：

+1 for this. Thanks all the contributors of Flink CDC for the great work.

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "Martijn Visser" <ma...@apache.org>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36
主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Hi Leonard,

+1 for this. I think the CDC connectors are a great result and example
of the Flink CDC community. Kudos to you all.

Best regards,

Martijn

On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com>
wrote:

Hi Leonard,

Thank you for the excellent work you and the team working on the CDC
connectors project have been doing so far. I am +1 of having them under
Flink's umbrella.

On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
wrote:

Big +1, thanks this will be a very useful addition to Flink.

Best

Etienne

Le 07/12/2023 à 09:26, Hang Ruan a écrit :
+1 for contributing CDC Connectors  to Apache Flink.

Best,
Hang

Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：

Cool, +1 for contributing CDC Connectors to Apache Flink.

Best,
Yuxin


Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：

Awesome! +1

Best regards,
Jing

On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<snuyanzin@gmail.com

wrote:

thanks for working on this and driving it

+1

On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
wrote:

This is incredibly exciting news, a big +1 for this.

Thank you for the fantastic work on Flink CDC. We have created
thousands
of
real-time integration jobs using Flink CDC connectors.


Best,
Feng

On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
gongzhongqiang@apache.org
wrote:

It's very exciting to hear the news.
+1 for adding CDC Connectors  to Apache Flink !


Best,
Zhongqiang

Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：

Dear Flink devs,


As you may have heard, we at Alibaba (Ververica) are planning
to
donate
CDC Connectors for the Apache Flink project
*[1]* to the Apache Flink community.



CDC Connectors for Apache Flink comprise a collection of source
connectors designed specifically for Apache Flink. These
connectors
*[2]*
enable the ingestion of changes from various databases using
Change
Data Capture (CDC), most of these CDC connectors are powered by
Debezium
*[3]*
. They support both the DataStream API and the Table/SQL API,
facilitating the reading of database snapshots and continuous
reading
of
transaction logs with exactly-once processing, even in the
event of
failures.


Additionally, in the latest version 3.0, we have introduced
many
long-awaited features. Starting from CDC version 3.0, we've
built a
Streaming ELT Framework available for streaming data
integration.
This
framework allows users to write their data synchronization logic
in a
simple YAML file, which will automatically be translated into a
Flink
DataStreaming job. It emphasizes optimizing the task submission
process
and
offers advanced functionalities such as whole database
synchronization,
merging sharded tables, and schema evolution
*[4]*.




I believe this initiative is a perfect match for both sides.
For
the
Flink community, it presents an opportunity to enhance Flink's
competitive
advantage in streaming data integration, promoting the healthy
growth
and
prosperity of the Apache Flink ecosystem. For the CDC Connectors
project,
becoming a sub-project of Apache Flink means being part of a
neutral
open-source community, which can attract a more diverse pool of
contributors.

Please note that the aforementioned points represent only some
of
our
motivations and vision for this donation. Specific future
operations
need
to be further discussed in this thread. For example, the
sub-project
name
after the donation; we hope to name it Flink-CDC
aiming to streaming data intergration through Apache Flink,
following the naming convention of Flink-ML; And this project
is
managed
by a total of 8 maintainers, including 3 Flink PMC members and 1
Flink
Committer. The remaining 4 maintainers are also highly active
contributors
to the Flink community, donating this project to the Flink
community
implies that their permissions might be reduced. Therefore, we
may
need
to
bring up this topic for further discussion within the Flink PMC.
Additionally, we need to discuss how to migrate existing users
and
documents. We have a user group of nearly 10,000 people and a
multi-version
documentation site need to migrate. We also need to plan for the
migration
of CI/CD processes and other specifics.


While there are many intricate details that require
implementation,
we
are committed to progressing and finalizing this donation
process.


Despite being Flink’s most active ecological project (as
evaluated
by
GitHub metrics), it also boasts a significant user base.
However, I
believe
it's essential to commence discussions on future operations only
after
the
community reaches a consensus on whether they desire this
donation.

Really looking forward to hear what you think!



Best,
Leonard (on behalf of the Flink CDC Connectors project
maintainers)
[1]https://github.com/ververica/flink-cdc-connectors
[2]



https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
[3]https://debezium.io
[4]



https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

--
Best regards,
Sergey




--
Best

ConradJam

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by ConradJam <ja...@gmail.com>.
+1 Best Idea . Thanks to all the contributors of Flink CDC for the great
work.

yuxia <lu...@alumni.sjtu.edu.cn> 于2023年12月7日周四 20:46写道：

> +1 for this. Thanks all the contributors of Flink CDC for the great work.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Martijn Visser" <ma...@apache.org>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36
> 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
>
> Hi Leonard,
>
> +1 for this. I think the CDC connectors are a great result and example
> of the Flink CDC community. Kudos to you all.
>
> Best regards,
>
> Martijn
>
> On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com>
> wrote:
> >
> > Hi Leonard,
> >
> > Thank you for the excellent work you and the team working on the CDC
> > connectors project have been doing so far. I am +1 of having them under
> > Flink's umbrella.
> >
> > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
> > wrote:
> >
> > > Big +1, thanks this will be a very useful addition to Flink.
> > >
> > > Best
> > >
> > > Etienne
> > >
> > > Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > > > +1 for contributing CDC Connectors  to Apache Flink.
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
> > > >
> > > >> Cool, +1 for contributing CDC Connectors to Apache Flink.
> > > >>
> > > >> Best,
> > > >> Yuxin
> > > >>
> > > >>
> > > >> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
> > > >>
> > > >>> Awesome! +1
> > > >>>
> > > >>> Best regards,
> > > >>> Jing
> > > >>>
> > > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<snuyanzin@gmail.com
> >
> > > >>> wrote:
> > > >>>
> > > >>>> thanks for working on this and driving it
> > > >>>>
> > > >>>> +1
> > > >>>>
> > > >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>>> This is incredibly exciting news, a big +1 for this.
> > > >>>>>
> > > >>>>> Thank you for the fantastic work on Flink CDC. We have created
> > > >>> thousands
> > > >>>> of
> > > >>>>> real-time integration jobs using Flink CDC connectors.
> > > >>>>>
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Feng
> > > >>>>>
> > > >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> > > >>> gongzhongqiang@apache.org
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> It's very exciting to hear the news.
> > > >>>>>> +1 for adding CDC Connectors  to Apache Flink !
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Zhongqiang
> > > >>>>>>
> > > >>>>>> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
> > > >>>>>>
> > > >>>>>>> Dear Flink devs,
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning
> to
> > > >>>> donate
> > > >>>>>> CDC Connectors for the Apache Flink project
> > > >>>>>>> *[1]* to the Apache Flink community.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> CDC Connectors for Apache Flink comprise a collection of source
> > > >>>>>> connectors designed specifically for Apache Flink. These
> connectors
> > > >>>>>>> *[2]*
> > > >>>>>>>   enable the ingestion of changes from various databases using
> > > >>> Change
> > > >>>>>> Data Capture (CDC), most of these CDC connectors are powered by
> > > >>>> Debezium
> > > >>>>>>> *[3]*
> > > >>>>>>> . They support both the DataStream API and the Table/SQL API,
> > > >>>>>> facilitating the reading of database snapshots and continuous
> > > >> reading
> > > >>>> of
> > > >>>>>> transaction logs with exactly-once processing, even in the
> event of
> > > >>>>>> failures.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Additionally, in the latest version 3.0, we have introduced
> many
> > > >>>>>> long-awaited features. Starting from CDC version 3.0, we've
> built a
> > > >>>>>> Streaming ELT Framework available for streaming data
> integration.
> > > >>> This
> > > >>>>>> framework allows users to write their data synchronization logic
> > > >> in a
> > > >>>>>> simple YAML file, which will automatically be translated into a
> > > >> Flink
> > > >>>>>> DataStreaming job. It emphasizes optimizing the task submission
> > > >>> process
> > > >>>>> and
> > > >>>>>> offers advanced functionalities such as whole database
> > > >>> synchronization,
> > > >>>>>> merging sharded tables, and schema evolution
> > > >>>>>>> *[4]*.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> I believe this initiative is a perfect match for both sides.
> For
> > > >>> the
> > > >>>>>> Flink community, it presents an opportunity to enhance Flink's
> > > >>>>> competitive
> > > >>>>>> advantage in streaming data integration, promoting the healthy
> > > >> growth
> > > >>>> and
> > > >>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > > >>>> project,
> > > >>>>>> becoming a sub-project of Apache Flink means being part of a
> > > >> neutral
> > > >>>>>> open-source community, which can attract a more diverse pool of
> > > >>>>>> contributors.
> > > >>>>>>>
> > > >>>>>>> Please note that the aforementioned points represent only some
> of
> > > >>> our
> > > >>>>>> motivations and vision for this donation. Specific future
> > > >> operations
> > > >>>> need
> > > >>>>>> to be further discussed in this thread. For example, the
> > > >> sub-project
> > > >>>> name
> > > >>>>>> after the donation; we hope to name it Flink-CDC
> > > >>>>>>> aiming to streaming data intergration through Apache Flink,
> > > >>>>>>> following the naming convention of Flink-ML; And this project
> is
> > > >>>>> managed
> > > >>>>>> by a total of 8 maintainers, including 3 Flink PMC members and 1
> > > >>> Flink
> > > >>>>>> Committer. The remaining 4 maintainers are also highly active
> > > >>>>> contributors
> > > >>>>>> to the Flink community, donating this project to the Flink
> > > >> community
> > > >>>>>> implies that their permissions might be reduced. Therefore, we
> may
> > > >>> need
> > > >>>>> to
> > > >>>>>> bring up this topic for further discussion within the Flink PMC.
> > > >>>>>> Additionally, we need to discuss how to migrate existing users
> and
> > > >>>>>> documents. We have a user group of nearly 10,000 people and a
> > > >>>>> multi-version
> > > >>>>>> documentation site need to migrate. We also need to plan for the
> > > >>>>> migration
> > > >>>>>> of CI/CD processes and other specifics.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> While there are many intricate details that require
> > > >> implementation,
> > > >>>> we
> > > >>>>>> are committed to progressing and finalizing this donation
> process.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Despite being Flink’s most active ecological project (as
> > > >> evaluated
> > > >>> by
> > > >>>>>> GitHub metrics), it also boasts a significant user base.
> However, I
> > > >>>>> believe
> > > >>>>>> it's essential to commence discussions on future operations only
> > > >>> after
> > > >>>>> the
> > > >>>>>> community reaches a consensus on whether they desire this
> donation.
> > > >>>>>>>
> > > >>>>>>> Really looking forward to hear what you think!
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Best,
> > > >>>>>>> Leonard (on behalf of the Flink CDC Connectors project
> > > >> maintainers)
> > > >>>>>>> [1]https://github.com/ververica/flink-cdc-connectors
> > > >>>>>>> [2]
> > > >>>>>>>
> > > >>
> > >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > >>>>>>> [3]https://debezium.io
> > > >>>>>>> [4]
> > > >>>>>>>
> > > >>
> > >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > >>>>
> > > >>>> --
> > > >>>> Best regards,
> > > >>>> Sergey
> > > >>>>
>


-- 
Best

ConradJam

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
+1 for this. Thanks all the contributors of Flink CDC for the great work.

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "Martijn Visser" <ma...@apache.org>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36
主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Hi Leonard,

+1 for this. I think the CDC connectors are a great result and example
of the Flink CDC community. Kudos to you all.

Best regards,

Martijn

On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com> wrote:
>
> Hi Leonard,
>
> Thank you for the excellent work you and the team working on the CDC
> connectors project have been doing so far. I am +1 of having them under
> Flink's umbrella.
>
> On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
> > Big +1, thanks this will be a very useful addition to Flink.
> >
> > Best
> >
> > Etienne
> >
> > Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > > +1 for contributing CDC Connectors  to Apache Flink.
> > >
> > > Best,
> > > Hang
> > >
> > > Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
> > >
> > >> Cool, +1 for contributing CDC Connectors to Apache Flink.
> > >>
> > >> Best,
> > >> Yuxin
> > >>
> > >>
> > >> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
> > >>
> > >>> Awesome! +1
> > >>>
> > >>> Best regards,
> > >>> Jing
> > >>>
> > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<sn...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> thanks for working on this and driving it
> > >>>>
> > >>>> +1
> > >>>>
> > >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> This is incredibly exciting news, a big +1 for this.
> > >>>>>
> > >>>>> Thank you for the fantastic work on Flink CDC. We have created
> > >>> thousands
> > >>>> of
> > >>>>> real-time integration jobs using Flink CDC connectors.
> > >>>>>
> > >>>>>
> > >>>>> Best,
> > >>>>> Feng
> > >>>>>
> > >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> > >>> gongzhongqiang@apache.org
> > >>>>> wrote:
> > >>>>>
> > >>>>>> It's very exciting to hear the news.
> > >>>>>> +1 for adding CDC Connectors  to Apache Flink !
> > >>>>>>
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Zhongqiang
> > >>>>>>
> > >>>>>> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
> > >>>>>>
> > >>>>>>> Dear Flink devs,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
> > >>>> donate
> > >>>>>> CDC Connectors for the Apache Flink project
> > >>>>>>> *[1]* to the Apache Flink community.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> CDC Connectors for Apache Flink comprise a collection of source
> > >>>>>> connectors designed specifically for Apache Flink. These connectors
> > >>>>>>> *[2]*
> > >>>>>>>   enable the ingestion of changes from various databases using
> > >>> Change
> > >>>>>> Data Capture (CDC), most of these CDC connectors are powered by
> > >>>> Debezium
> > >>>>>>> *[3]*
> > >>>>>>> . They support both the DataStream API and the Table/SQL API,
> > >>>>>> facilitating the reading of database snapshots and continuous
> > >> reading
> > >>>> of
> > >>>>>> transaction logs with exactly-once processing, even in the event of
> > >>>>>> failures.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Additionally, in the latest version 3.0, we have introduced many
> > >>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
> > >>>>>> Streaming ELT Framework available for streaming data integration.
> > >>> This
> > >>>>>> framework allows users to write their data synchronization logic
> > >> in a
> > >>>>>> simple YAML file, which will automatically be translated into a
> > >> Flink
> > >>>>>> DataStreaming job. It emphasizes optimizing the task submission
> > >>> process
> > >>>>> and
> > >>>>>> offers advanced functionalities such as whole database
> > >>> synchronization,
> > >>>>>> merging sharded tables, and schema evolution
> > >>>>>>> *[4]*.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I believe this initiative is a perfect match for both sides. For
> > >>> the
> > >>>>>> Flink community, it presents an opportunity to enhance Flink's
> > >>>>> competitive
> > >>>>>> advantage in streaming data integration, promoting the healthy
> > >> growth
> > >>>> and
> > >>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > >>>> project,
> > >>>>>> becoming a sub-project of Apache Flink means being part of a
> > >> neutral
> > >>>>>> open-source community, which can attract a more diverse pool of
> > >>>>>> contributors.
> > >>>>>>>
> > >>>>>>> Please note that the aforementioned points represent only some of
> > >>> our
> > >>>>>> motivations and vision for this donation. Specific future
> > >> operations
> > >>>> need
> > >>>>>> to be further discussed in this thread. For example, the
> > >> sub-project
> > >>>> name
> > >>>>>> after the donation; we hope to name it Flink-CDC
> > >>>>>>> aiming to streaming data intergration through Apache Flink,
> > >>>>>>> following the naming convention of Flink-ML; And this project is
> > >>>>> managed
> > >>>>>> by a total of 8 maintainers, including 3 Flink PMC members and 1
> > >>> Flink
> > >>>>>> Committer. The remaining 4 maintainers are also highly active
> > >>>>> contributors
> > >>>>>> to the Flink community, donating this project to the Flink
> > >> community
> > >>>>>> implies that their permissions might be reduced. Therefore, we may
> > >>> need
> > >>>>> to
> > >>>>>> bring up this topic for further discussion within the Flink PMC.
> > >>>>>> Additionally, we need to discuss how to migrate existing users and
> > >>>>>> documents. We have a user group of nearly 10,000 people and a
> > >>>>> multi-version
> > >>>>>> documentation site need to migrate. We also need to plan for the
> > >>>>> migration
> > >>>>>> of CI/CD processes and other specifics.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> While there are many intricate details that require
> > >> implementation,
> > >>>> we
> > >>>>>> are committed to progressing and finalizing this donation process.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Despite being Flink’s most active ecological project (as
> > >> evaluated
> > >>> by
> > >>>>>> GitHub metrics), it also boasts a significant user base. However, I
> > >>>>> believe
> > >>>>>> it's essential to commence discussions on future operations only
> > >>> after
> > >>>>> the
> > >>>>>> community reaches a consensus on whether they desire this donation.
> > >>>>>>>
> > >>>>>>> Really looking forward to hear what you think!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Leonard (on behalf of the Flink CDC Connectors project
> > >> maintainers)
> > >>>>>>> [1]https://github.com/ververica/flink-cdc-connectors
> > >>>>>>> [2]
> > >>>>>>>
> > >>
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > >>>>>>> [3]https://debezium.io
> > >>>>>>> [4]
> > >>>>>>>
> > >>
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Sergey
> > >>>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Martijn Visser <ma...@apache.org>.
Hi Leonard,

+1 for this. I think the CDC connectors are a great result and example
of the Flink CDC community. Kudos to you all.

Best regards,

Martijn

On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi <ba...@gmail.com> wrote:
>
> Hi Leonard,
>
> Thank you for the excellent work you and the team working on the CDC
> connectors project have been doing so far. I am +1 of having them under
> Flink's umbrella.
>
> On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
> > Big +1, thanks this will be a very useful addition to Flink.
> >
> > Best
> >
> > Etienne
> >
> > Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > > +1 for contributing CDC Connectors  to Apache Flink.
> > >
> > > Best,
> > > Hang
> > >
> > > Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
> > >
> > >> Cool, +1 for contributing CDC Connectors to Apache Flink.
> > >>
> > >> Best,
> > >> Yuxin
> > >>
> > >>
> > >> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
> > >>
> > >>> Awesome! +1
> > >>>
> > >>> Best regards,
> > >>> Jing
> > >>>
> > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<sn...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> thanks for working on this and driving it
> > >>>>
> > >>>> +1
> > >>>>
> > >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> This is incredibly exciting news, a big +1 for this.
> > >>>>>
> > >>>>> Thank you for the fantastic work on Flink CDC. We have created
> > >>> thousands
> > >>>> of
> > >>>>> real-time integration jobs using Flink CDC connectors.
> > >>>>>
> > >>>>>
> > >>>>> Best,
> > >>>>> Feng
> > >>>>>
> > >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> > >>> gongzhongqiang@apache.org
> > >>>>> wrote:
> > >>>>>
> > >>>>>> It's very exciting to hear the news.
> > >>>>>> +1 for adding CDC Connectors  to Apache Flink !
> > >>>>>>
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Zhongqiang
> > >>>>>>
> > >>>>>> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
> > >>>>>>
> > >>>>>>> Dear Flink devs,
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
> > >>>> donate
> > >>>>>> CDC Connectors for the Apache Flink project
> > >>>>>>> *[1]* to the Apache Flink community.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> CDC Connectors for Apache Flink comprise a collection of source
> > >>>>>> connectors designed specifically for Apache Flink. These connectors
> > >>>>>>> *[2]*
> > >>>>>>>   enable the ingestion of changes from various databases using
> > >>> Change
> > >>>>>> Data Capture (CDC), most of these CDC connectors are powered by
> > >>>> Debezium
> > >>>>>>> *[3]*
> > >>>>>>> . They support both the DataStream API and the Table/SQL API,
> > >>>>>> facilitating the reading of database snapshots and continuous
> > >> reading
> > >>>> of
> > >>>>>> transaction logs with exactly-once processing, even in the event of
> > >>>>>> failures.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Additionally, in the latest version 3.0, we have introduced many
> > >>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
> > >>>>>> Streaming ELT Framework available for streaming data integration.
> > >>> This
> > >>>>>> framework allows users to write their data synchronization logic
> > >> in a
> > >>>>>> simple YAML file, which will automatically be translated into a
> > >> Flink
> > >>>>>> DataStreaming job. It emphasizes optimizing the task submission
> > >>> process
> > >>>>> and
> > >>>>>> offers advanced functionalities such as whole database
> > >>> synchronization,
> > >>>>>> merging sharded tables, and schema evolution
> > >>>>>>> *[4]*.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I believe this initiative is a perfect match for both sides. For
> > >>> the
> > >>>>>> Flink community, it presents an opportunity to enhance Flink's
> > >>>>> competitive
> > >>>>>> advantage in streaming data integration, promoting the healthy
> > >> growth
> > >>>> and
> > >>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > >>>> project,
> > >>>>>> becoming a sub-project of Apache Flink means being part of a
> > >> neutral
> > >>>>>> open-source community, which can attract a more diverse pool of
> > >>>>>> contributors.
> > >>>>>>>
> > >>>>>>> Please note that the aforementioned points represent only some of
> > >>> our
> > >>>>>> motivations and vision for this donation. Specific future
> > >> operations
> > >>>> need
> > >>>>>> to be further discussed in this thread. For example, the
> > >> sub-project
> > >>>> name
> > >>>>>> after the donation; we hope to name it Flink-CDC
> > >>>>>>> aiming to streaming data intergration through Apache Flink,
> > >>>>>>> following the naming convention of Flink-ML; And this project is
> > >>>>> managed
> > >>>>>> by a total of 8 maintainers, including 3 Flink PMC members and 1
> > >>> Flink
> > >>>>>> Committer. The remaining 4 maintainers are also highly active
> > >>>>> contributors
> > >>>>>> to the Flink community, donating this project to the Flink
> > >> community
> > >>>>>> implies that their permissions might be reduced. Therefore, we may
> > >>> need
> > >>>>> to
> > >>>>>> bring up this topic for further discussion within the Flink PMC.
> > >>>>>> Additionally, we need to discuss how to migrate existing users and
> > >>>>>> documents. We have a user group of nearly 10,000 people and a
> > >>>>> multi-version
> > >>>>>> documentation site need to migrate. We also need to plan for the
> > >>>>> migration
> > >>>>>> of CI/CD processes and other specifics.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> While there are many intricate details that require
> > >> implementation,
> > >>>> we
> > >>>>>> are committed to progressing and finalizing this donation process.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Despite being Flink’s most active ecological project (as
> > >> evaluated
> > >>> by
> > >>>>>> GitHub metrics), it also boasts a significant user base. However, I
> > >>>>> believe
> > >>>>>> it's essential to commence discussions on future operations only
> > >>> after
> > >>>>> the
> > >>>>>> community reaches a consensus on whether they desire this donation.
> > >>>>>>>
> > >>>>>>> Really looking forward to hear what you think!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Leonard (on behalf of the Flink CDC Connectors project
> > >> maintainers)
> > >>>>>>> [1]https://github.com/ververica/flink-cdc-connectors
> > >>>>>>> [2]
> > >>>>>>>
> > >>
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > >>>>>>> [3]https://debezium.io
> > >>>>>>> [4]
> > >>>>>>>
> > >>
> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>> Sergey
> > >>>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Márton Balassi <ba...@gmail.com>.
Hi Leonard,

Thank you for the excellent work you and the team working on the CDC
connectors project have been doing so far. I am +1 of having them under
Flink's umbrella.

On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot <ec...@apache.org>
wrote:

> Big +1, thanks this will be a very useful addition to Flink.
>
> Best
>
> Etienne
>
> Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > +1 for contributing CDC Connectors  to Apache Flink.
> >
> > Best,
> > Hang
> >
> > Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
> >
> >> Cool, +1 for contributing CDC Connectors to Apache Flink.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
> >>
> >>> Awesome! +1
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<sn...@gmail.com>
> >>> wrote:
> >>>
> >>>> thanks for working on this and driving it
> >>>>
> >>>> +1
> >>>>
> >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>
> wrote:
> >>>>
> >>>>> This is incredibly exciting news, a big +1 for this.
> >>>>>
> >>>>> Thank you for the fantastic work on Flink CDC. We have created
> >>> thousands
> >>>> of
> >>>>> real-time integration jobs using Flink CDC connectors.
> >>>>>
> >>>>>
> >>>>> Best,
> >>>>> Feng
> >>>>>
> >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> >>> gongzhongqiang@apache.org
> >>>>> wrote:
> >>>>>
> >>>>>> It's very exciting to hear the news.
> >>>>>> +1 for adding CDC Connectors  to Apache Flink !
> >>>>>>
> >>>>>>
> >>>>>> Best,
> >>>>>> Zhongqiang
> >>>>>>
> >>>>>> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
> >>>>>>
> >>>>>>> Dear Flink devs,
> >>>>>>>
> >>>>>>>
> >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
> >>>> donate
> >>>>>> CDC Connectors for the Apache Flink project
> >>>>>>> *[1]* to the Apache Flink community.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> CDC Connectors for Apache Flink comprise a collection of source
> >>>>>> connectors designed specifically for Apache Flink. These connectors
> >>>>>>> *[2]*
> >>>>>>>   enable the ingestion of changes from various databases using
> >>> Change
> >>>>>> Data Capture (CDC), most of these CDC connectors are powered by
> >>>> Debezium
> >>>>>>> *[3]*
> >>>>>>> . They support both the DataStream API and the Table/SQL API,
> >>>>>> facilitating the reading of database snapshots and continuous
> >> reading
> >>>> of
> >>>>>> transaction logs with exactly-once processing, even in the event of
> >>>>>> failures.
> >>>>>>>
> >>>>>>>
> >>>>>>> Additionally, in the latest version 3.0, we have introduced many
> >>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
> >>>>>> Streaming ELT Framework available for streaming data integration.
> >>> This
> >>>>>> framework allows users to write their data synchronization logic
> >> in a
> >>>>>> simple YAML file, which will automatically be translated into a
> >> Flink
> >>>>>> DataStreaming job. It emphasizes optimizing the task submission
> >>> process
> >>>>> and
> >>>>>> offers advanced functionalities such as whole database
> >>> synchronization,
> >>>>>> merging sharded tables, and schema evolution
> >>>>>>> *[4]*.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I believe this initiative is a perfect match for both sides. For
> >>> the
> >>>>>> Flink community, it presents an opportunity to enhance Flink's
> >>>>> competitive
> >>>>>> advantage in streaming data integration, promoting the healthy
> >> growth
> >>>> and
> >>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> >>>> project,
> >>>>>> becoming a sub-project of Apache Flink means being part of a
> >> neutral
> >>>>>> open-source community, which can attract a more diverse pool of
> >>>>>> contributors.
> >>>>>>>
> >>>>>>> Please note that the aforementioned points represent only some of
> >>> our
> >>>>>> motivations and vision for this donation. Specific future
> >> operations
> >>>> need
> >>>>>> to be further discussed in this thread. For example, the
> >> sub-project
> >>>> name
> >>>>>> after the donation; we hope to name it Flink-CDC
> >>>>>>> aiming to streaming data intergration through Apache Flink,
> >>>>>>> following the naming convention of Flink-ML; And this project is
> >>>>> managed
> >>>>>> by a total of 8 maintainers, including 3 Flink PMC members and 1
> >>> Flink
> >>>>>> Committer. The remaining 4 maintainers are also highly active
> >>>>> contributors
> >>>>>> to the Flink community, donating this project to the Flink
> >> community
> >>>>>> implies that their permissions might be reduced. Therefore, we may
> >>> need
> >>>>> to
> >>>>>> bring up this topic for further discussion within the Flink PMC.
> >>>>>> Additionally, we need to discuss how to migrate existing users and
> >>>>>> documents. We have a user group of nearly 10,000 people and a
> >>>>> multi-version
> >>>>>> documentation site need to migrate. We also need to plan for the
> >>>>> migration
> >>>>>> of CI/CD processes and other specifics.
> >>>>>>>
> >>>>>>>
> >>>>>>> While there are many intricate details that require
> >> implementation,
> >>>> we
> >>>>>> are committed to progressing and finalizing this donation process.
> >>>>>>>
> >>>>>>>
> >>>>>>> Despite being Flink’s most active ecological project (as
> >> evaluated
> >>> by
> >>>>>> GitHub metrics), it also boasts a significant user base. However, I
> >>>>> believe
> >>>>>> it's essential to commence discussions on future operations only
> >>> after
> >>>>> the
> >>>>>> community reaches a consensus on whether they desire this donation.
> >>>>>>>
> >>>>>>> Really looking forward to hear what you think!
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Leonard (on behalf of the Flink CDC Connectors project
> >> maintainers)
> >>>>>>> [1]https://github.com/ververica/flink-cdc-connectors
> >>>>>>> [2]
> >>>>>>>
> >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> >>>>>>> [3]https://debezium.io
> >>>>>>> [4]
> >>>>>>>
> >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Sergey
> >>>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Etienne Chauchot <ec...@apache.org>.
Big +1, thanks this will be a very useful addition to Flink.

Best

Etienne

Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> +1 for contributing CDC Connectors  to Apache Flink.
>
> Best,
> Hang
>
> Yuxin Tan<ta...@gmail.com>  于2023年12月7日周四 16:05写道：
>
>> Cool, +1 for contributing CDC Connectors to Apache Flink.
>>
>> Best,
>> Yuxin
>>
>>
>> Jing Ge<ji...@ververica.com.invalid>  于2023年12月7日周四 15:43写道：
>>
>>> Awesome! +1
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<sn...@gmail.com>
>>> wrote:
>>>
>>>> thanks for working on this and driving it
>>>>
>>>> +1
>>>>
>>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<ji...@gmail.com>  wrote:
>>>>
>>>>> This is incredibly exciting news, a big +1 for this.
>>>>>
>>>>> Thank you for the fantastic work on Flink CDC. We have created
>>> thousands
>>>> of
>>>>> real-time integration jobs using Flink CDC connectors.
>>>>>
>>>>>
>>>>> Best,
>>>>> Feng
>>>>>
>>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
>>> gongzhongqiang@apache.org
>>>>> wrote:
>>>>>
>>>>>> It's very exciting to hear the news.
>>>>>> +1 for adding CDC Connectors  to Apache Flink !
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Zhongqiang
>>>>>>
>>>>>> Leonard Xu<xb...@gmail.com>  于2023年12月7日周四 11:25写道：
>>>>>>
>>>>>>> Dear Flink devs,
>>>>>>>
>>>>>>>
>>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
>>>> donate
>>>>>> CDC Connectors for the Apache Flink project
>>>>>>> *[1]* to the Apache Flink community.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CDC Connectors for Apache Flink comprise a collection of source
>>>>>> connectors designed specifically for Apache Flink. These connectors
>>>>>>> *[2]*
>>>>>>>   enable the ingestion of changes from various databases using
>>> Change
>>>>>> Data Capture (CDC), most of these CDC connectors are powered by
>>>> Debezium
>>>>>>> *[3]*
>>>>>>> . They support both the DataStream API and the Table/SQL API,
>>>>>> facilitating the reading of database snapshots and continuous
>> reading
>>>> of
>>>>>> transaction logs with exactly-once processing, even in the event of
>>>>>> failures.
>>>>>>>
>>>>>>>
>>>>>>> Additionally, in the latest version 3.0, we have introduced many
>>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
>>>>>> Streaming ELT Framework available for streaming data integration.
>>> This
>>>>>> framework allows users to write their data synchronization logic
>> in a
>>>>>> simple YAML file, which will automatically be translated into a
>> Flink
>>>>>> DataStreaming job. It emphasizes optimizing the task submission
>>> process
>>>>> and
>>>>>> offers advanced functionalities such as whole database
>>> synchronization,
>>>>>> merging sharded tables, and schema evolution
>>>>>>> *[4]*.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I believe this initiative is a perfect match for both sides. For
>>> the
>>>>>> Flink community, it presents an opportunity to enhance Flink's
>>>>> competitive
>>>>>> advantage in streaming data integration, promoting the healthy
>> growth
>>>> and
>>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
>>>> project,
>>>>>> becoming a sub-project of Apache Flink means being part of a
>> neutral
>>>>>> open-source community, which can attract a more diverse pool of
>>>>>> contributors.
>>>>>>>
>>>>>>> Please note that the aforementioned points represent only some of
>>> our
>>>>>> motivations and vision for this donation. Specific future
>> operations
>>>> need
>>>>>> to be further discussed in this thread. For example, the
>> sub-project
>>>> name
>>>>>> after the donation; we hope to name it Flink-CDC
>>>>>>> aiming to streaming data intergration through Apache Flink,
>>>>>>> following the naming convention of Flink-ML; And this project is
>>>>> managed
>>>>>> by a total of 8 maintainers, including 3 Flink PMC members and 1
>>> Flink
>>>>>> Committer. The remaining 4 maintainers are also highly active
>>>>> contributors
>>>>>> to the Flink community, donating this project to the Flink
>> community
>>>>>> implies that their permissions might be reduced. Therefore, we may
>>> need
>>>>> to
>>>>>> bring up this topic for further discussion within the Flink PMC.
>>>>>> Additionally, we need to discuss how to migrate existing users and
>>>>>> documents. We have a user group of nearly 10,000 people and a
>>>>> multi-version
>>>>>> documentation site need to migrate. We also need to plan for the
>>>>> migration
>>>>>> of CI/CD processes and other specifics.
>>>>>>>
>>>>>>>
>>>>>>> While there are many intricate details that require
>> implementation,
>>>> we
>>>>>> are committed to progressing and finalizing this donation process.
>>>>>>>
>>>>>>>
>>>>>>> Despite being Flink’s most active ecological project (as
>> evaluated
>>> by
>>>>>> GitHub metrics), it also boasts a significant user base. However, I
>>>>> believe
>>>>>> it's essential to commence discussions on future operations only
>>> after
>>>>> the
>>>>>> community reaches a consensus on whether they desire this donation.
>>>>>>>
>>>>>>> Really looking forward to hear what you think!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Leonard (on behalf of the Flink CDC Connectors project
>> maintainers)
>>>>>>> [1]https://github.com/ververica/flink-cdc-connectors
>>>>>>> [2]
>>>>>>>
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>>>>>>> [3]https://debezium.io
>>>>>>> [4]
>>>>>>>
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>>>
>>>> --
>>>> Best regards,
>>>> Sergey
>>>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Hang Ruan <ru...@gmail.com>.
+1 for contributing CDC Connectors  to Apache Flink.

Best,
Hang

Yuxin Tan <ta...@gmail.com> 于2023年12月7日周四 16:05写道：

> Cool, +1 for contributing CDC Connectors to Apache Flink.
>
> Best,
> Yuxin
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年12月7日周四 15:43写道：
>
> > Awesome! +1
> >
> > Best regards,
> > Jing
> >
> > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin <sn...@gmail.com>
> > wrote:
> >
> > > thanks for working on this and driving it
> > >
> > > +1
> > >
> > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin <ji...@gmail.com> wrote:
> > >
> > > > This is incredibly exciting news, a big +1 for this.
> > > >
> > > > Thank you for the fantastic work on Flink CDC. We have created
> > thousands
> > > of
> > > > real-time integration jobs using Flink CDC connectors.
> > > >
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> > gongzhongqiang@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > It's very exciting to hear the news.
> > > > > +1 for adding CDC Connectors  to Apache Flink !
> > > > >
> > > > >
> > > > > Best,
> > > > > Zhongqiang
> > > > >
> > > > > Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> > > > >
> > > > > > Dear Flink devs,
> > > > > >
> > > > > >
> > > > > > As you may have heard, we at Alibaba (Ververica) are planning to
> > > donate
> > > > > CDC Connectors for the Apache Flink project
> > > > > > *[1]* to the Apache Flink community.
> > > > > >
> > > > > >
> > > > > >
> > > > > > CDC Connectors for Apache Flink comprise a collection of source
> > > > > connectors designed specifically for Apache Flink. These connectors
> > > > > > *[2]*
> > > > > >  enable the ingestion of changes from various databases using
> > Change
> > > > > Data Capture (CDC), most of these CDC connectors are powered by
> > > Debezium
> > > > > > *[3]*
> > > > > > . They support both the DataStream API and the Table/SQL API,
> > > > > facilitating the reading of database snapshots and continuous
> reading
> > > of
> > > > > transaction logs with exactly-once processing, even in the event of
> > > > > failures.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Additionally, in the latest version 3.0, we have introduced many
> > > > > long-awaited features. Starting from CDC version 3.0, we've built a
> > > > > Streaming ELT Framework available for streaming data integration.
> > This
> > > > > framework allows users to write their data synchronization logic
> in a
> > > > > simple YAML file, which will automatically be translated into a
> Flink
> > > > > DataStreaming job. It emphasizes optimizing the task submission
> > process
> > > > and
> > > > > offers advanced functionalities such as whole database
> > synchronization,
> > > > > merging sharded tables, and schema evolution
> > > > > > *[4]*.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > I believe this initiative is a perfect match for both sides. For
> > the
> > > > > Flink community, it presents an opportunity to enhance Flink's
> > > > competitive
> > > > > advantage in streaming data integration, promoting the healthy
> growth
> > > and
> > > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > > project,
> > > > > becoming a sub-project of Apache Flink means being part of a
> neutral
> > > > > open-source community, which can attract a more diverse pool of
> > > > > contributors.
> > > > > >
> > > > > >
> > > > > > Please note that the aforementioned points represent only some of
> > our
> > > > > motivations and vision for this donation. Specific future
> operations
> > > need
> > > > > to be further discussed in this thread. For example, the
> sub-project
> > > name
> > > > > after the donation; we hope to name it Flink-CDC
> > > > > > aiming to streaming data intergration through Apache Flink,
> > > > > > following the naming convention of Flink-ML; And this project is
> > > > managed
> > > > > by a total of 8 maintainers, including 3 Flink PMC members and 1
> > Flink
> > > > > Committer. The remaining 4 maintainers are also highly active
> > > > contributors
> > > > > to the Flink community, donating this project to the Flink
> community
> > > > > implies that their permissions might be reduced. Therefore, we may
> > need
> > > > to
> > > > > bring up this topic for further discussion within the Flink PMC.
> > > > > Additionally, we need to discuss how to migrate existing users and
> > > > > documents. We have a user group of nearly 10,000 people and a
> > > > multi-version
> > > > > documentation site need to migrate. We also need to plan for the
> > > > migration
> > > > > of CI/CD processes and other specifics.
> > > > > >
> > > > > >
> > > > > >
> > > > > > While there are many intricate details that require
> implementation,
> > > we
> > > > > are committed to progressing and finalizing this donation process.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Despite being Flink’s most active ecological project (as
> evaluated
> > by
> > > > > GitHub metrics), it also boasts a significant user base. However, I
> > > > believe
> > > > > it's essential to commence discussions on future operations only
> > after
> > > > the
> > > > > community reaches a consensus on whether they desire this donation.
> > > > > >
> > > > > >
> > > > > > Really looking forward to hear what you think!
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Leonard (on behalf of the Flink CDC Connectors project
> maintainers)
> > > > > >
> > > > > > [1] https://github.com/ververica/flink-cdc-connectors
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > > > > [3] https://debezium.io
> > > > > > [4]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> > >
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Yuxin Tan <ta...@gmail.com>.
Cool, +1 for contributing CDC Connectors to Apache Flink.

Best,
Yuxin


Jing Ge <ji...@ververica.com.invalid> 于2023年12月7日周四 15:43写道：

> Awesome! +1
>
> Best regards,
> Jing
>
> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin <sn...@gmail.com>
> wrote:
>
> > thanks for working on this and driving it
> >
> > +1
> >
> > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin <ji...@gmail.com> wrote:
> >
> > > This is incredibly exciting news, a big +1 for this.
> > >
> > > Thank you for the fantastic work on Flink CDC. We have created
> thousands
> > of
> > > real-time integration jobs using Flink CDC connectors.
> > >
> > >
> > > Best,
> > > Feng
> > >
> > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> gongzhongqiang@apache.org
> > >
> > > wrote:
> > >
> > > > It's very exciting to hear the news.
> > > > +1 for adding CDC Connectors  to Apache Flink !
> > > >
> > > >
> > > > Best,
> > > > Zhongqiang
> > > >
> > > > Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> > > >
> > > > > Dear Flink devs,
> > > > >
> > > > >
> > > > > As you may have heard, we at Alibaba (Ververica) are planning to
> > donate
> > > > CDC Connectors for the Apache Flink project
> > > > > *[1]* to the Apache Flink community.
> > > > >
> > > > >
> > > > >
> > > > > CDC Connectors for Apache Flink comprise a collection of source
> > > > connectors designed specifically for Apache Flink. These connectors
> > > > > *[2]*
> > > > >  enable the ingestion of changes from various databases using
> Change
> > > > Data Capture (CDC), most of these CDC connectors are powered by
> > Debezium
> > > > > *[3]*
> > > > > . They support both the DataStream API and the Table/SQL API,
> > > > facilitating the reading of database snapshots and continuous reading
> > of
> > > > transaction logs with exactly-once processing, even in the event of
> > > > failures.
> > > > >
> > > > >
> > > > >
> > > > > Additionally, in the latest version 3.0, we have introduced many
> > > > long-awaited features. Starting from CDC version 3.0, we've built a
> > > > Streaming ELT Framework available for streaming data integration.
> This
> > > > framework allows users to write their data synchronization logic in a
> > > > simple YAML file, which will automatically be translated into a Flink
> > > > DataStreaming job. It emphasizes optimizing the task submission
> process
> > > and
> > > > offers advanced functionalities such as whole database
> synchronization,
> > > > merging sharded tables, and schema evolution
> > > > > *[4]*.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I believe this initiative is a perfect match for both sides. For
> the
> > > > Flink community, it presents an opportunity to enhance Flink's
> > > competitive
> > > > advantage in streaming data integration, promoting the healthy growth
> > and
> > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > project,
> > > > becoming a sub-project of Apache Flink means being part of a neutral
> > > > open-source community, which can attract a more diverse pool of
> > > > contributors.
> > > > >
> > > > >
> > > > > Please note that the aforementioned points represent only some of
> our
> > > > motivations and vision for this donation. Specific future operations
> > need
> > > > to be further discussed in this thread. For example, the sub-project
> > name
> > > > after the donation; we hope to name it Flink-CDC
> > > > > aiming to streaming data intergration through Apache Flink,
> > > > > following the naming convention of Flink-ML; And this project is
> > > managed
> > > > by a total of 8 maintainers, including 3 Flink PMC members and 1
> Flink
> > > > Committer. The remaining 4 maintainers are also highly active
> > > contributors
> > > > to the Flink community, donating this project to the Flink community
> > > > implies that their permissions might be reduced. Therefore, we may
> need
> > > to
> > > > bring up this topic for further discussion within the Flink PMC.
> > > > Additionally, we need to discuss how to migrate existing users and
> > > > documents. We have a user group of nearly 10,000 people and a
> > > multi-version
> > > > documentation site need to migrate. We also need to plan for the
> > > migration
> > > > of CI/CD processes and other specifics.
> > > > >
> > > > >
> > > > >
> > > > > While there are many intricate details that require implementation,
> > we
> > > > are committed to progressing and finalizing this donation process.
> > > > >
> > > > >
> > > > >
> > > > > Despite being Flink’s most active ecological project (as evaluated
> by
> > > > GitHub metrics), it also boasts a significant user base. However, I
> > > believe
> > > > it's essential to commence discussions on future operations only
> after
> > > the
> > > > community reaches a consensus on whether they desire this donation.
> > > > >
> > > > >
> > > > > Really looking forward to hear what you think!
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > > > >
> > > > > [1] https://github.com/ververica/flink-cdc-connectors
> > > > > [2]
> > > > >
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > > > [3] https://debezium.io
> > > > > [4]
> > > > >
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Awesome! +1

Best regards,
Jing

On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin <sn...@gmail.com> wrote:

> thanks for working on this and driving it
>
> +1
>
> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin <ji...@gmail.com> wrote:
>
> > This is incredibly exciting news, a big +1 for this.
> >
> > Thank you for the fantastic work on Flink CDC. We have created thousands
> of
> > real-time integration jobs using Flink CDC connectors.
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <gongzhongqiang@apache.org
> >
> > wrote:
> >
> > > It's very exciting to hear the news.
> > > +1 for adding CDC Connectors  to Apache Flink !
> > >
> > >
> > > Best,
> > > Zhongqiang
> > >
> > > Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> > >
> > > > Dear Flink devs,
> > > >
> > > >
> > > > As you may have heard, we at Alibaba (Ververica) are planning to
> donate
> > > CDC Connectors for the Apache Flink project
> > > > *[1]* to the Apache Flink community.
> > > >
> > > >
> > > >
> > > > CDC Connectors for Apache Flink comprise a collection of source
> > > connectors designed specifically for Apache Flink. These connectors
> > > > *[2]*
> > > >  enable the ingestion of changes from various databases using Change
> > > Data Capture (CDC), most of these CDC connectors are powered by
> Debezium
> > > > *[3]*
> > > > . They support both the DataStream API and the Table/SQL API,
> > > facilitating the reading of database snapshots and continuous reading
> of
> > > transaction logs with exactly-once processing, even in the event of
> > > failures.
> > > >
> > > >
> > > >
> > > > Additionally, in the latest version 3.0, we have introduced many
> > > long-awaited features. Starting from CDC version 3.0, we've built a
> > > Streaming ELT Framework available for streaming data integration. This
> > > framework allows users to write their data synchronization logic in a
> > > simple YAML file, which will automatically be translated into a Flink
> > > DataStreaming job. It emphasizes optimizing the task submission process
> > and
> > > offers advanced functionalities such as whole database synchronization,
> > > merging sharded tables, and schema evolution
> > > > *[4]*.
> > > >
> > > >
> > > >
> > > >
> > > > I believe this initiative is a perfect match for both sides. For the
> > > Flink community, it presents an opportunity to enhance Flink's
> > competitive
> > > advantage in streaming data integration, promoting the healthy growth
> and
> > > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> project,
> > > becoming a sub-project of Apache Flink means being part of a neutral
> > > open-source community, which can attract a more diverse pool of
> > > contributors.
> > > >
> > > >
> > > > Please note that the aforementioned points represent only some of our
> > > motivations and vision for this donation. Specific future operations
> need
> > > to be further discussed in this thread. For example, the sub-project
> name
> > > after the donation; we hope to name it Flink-CDC
> > > > aiming to streaming data intergration through Apache Flink,
> > > > following the naming convention of Flink-ML; And this project is
> > managed
> > > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink
> > > Committer. The remaining 4 maintainers are also highly active
> > contributors
> > > to the Flink community, donating this project to the Flink community
> > > implies that their permissions might be reduced. Therefore, we may need
> > to
> > > bring up this topic for further discussion within the Flink PMC.
> > > Additionally, we need to discuss how to migrate existing users and
> > > documents. We have a user group of nearly 10,000 people and a
> > multi-version
> > > documentation site need to migrate. We also need to plan for the
> > migration
> > > of CI/CD processes and other specifics.
> > > >
> > > >
> > > >
> > > > While there are many intricate details that require implementation,
> we
> > > are committed to progressing and finalizing this donation process.
> > > >
> > > >
> > > >
> > > > Despite being Flink’s most active ecological project (as evaluated by
> > > GitHub metrics), it also boasts a significant user base. However, I
> > believe
> > > it's essential to commence discussions on future operations only after
> > the
> > > community reaches a consensus on whether they desire this donation.
> > > >
> > > >
> > > > Really looking forward to hear what you think!
> > > >
> > > >
> > > >
> > > > Best,
> > > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > > >
> > > > [1] https://github.com/ververica/flink-cdc-connectors
> > > > [2]
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > > [3] https://debezium.io
> > > > [4]
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Sergey Nuyanzin <sn...@gmail.com>.
thanks for working on this and driving it

+1

On Thu, Dec 7, 2023 at 7:26 AM Feng Jin <ji...@gmail.com> wrote:

> This is incredibly exciting news, a big +1 for this.
>
> Thank you for the fantastic work on Flink CDC. We have created thousands of
> real-time integration jobs using Flink CDC connectors.
>
>
> Best,
> Feng
>
> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <go...@apache.org>
> wrote:
>
> > It's very exciting to hear the news.
> > +1 for adding CDC Connectors  to Apache Flink !
> >
> >
> > Best,
> > Zhongqiang
> >
> > Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> >
> > > Dear Flink devs,
> > >
> > >
> > > As you may have heard, we at Alibaba (Ververica) are planning to donate
> > CDC Connectors for the Apache Flink project
> > > *[1]* to the Apache Flink community.
> > >
> > >
> > >
> > > CDC Connectors for Apache Flink comprise a collection of source
> > connectors designed specifically for Apache Flink. These connectors
> > > *[2]*
> > >  enable the ingestion of changes from various databases using Change
> > Data Capture (CDC), most of these CDC connectors are powered by Debezium
> > > *[3]*
> > > . They support both the DataStream API and the Table/SQL API,
> > facilitating the reading of database snapshots and continuous reading of
> > transaction logs with exactly-once processing, even in the event of
> > failures.
> > >
> > >
> > >
> > > Additionally, in the latest version 3.0, we have introduced many
> > long-awaited features. Starting from CDC version 3.0, we've built a
> > Streaming ELT Framework available for streaming data integration. This
> > framework allows users to write their data synchronization logic in a
> > simple YAML file, which will automatically be translated into a Flink
> > DataStreaming job. It emphasizes optimizing the task submission process
> and
> > offers advanced functionalities such as whole database synchronization,
> > merging sharded tables, and schema evolution
> > > *[4]*.
> > >
> > >
> > >
> > >
> > > I believe this initiative is a perfect match for both sides. For the
> > Flink community, it presents an opportunity to enhance Flink's
> competitive
> > advantage in streaming data integration, promoting the healthy growth and
> > prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> > becoming a sub-project of Apache Flink means being part of a neutral
> > open-source community, which can attract a more diverse pool of
> > contributors.
> > >
> > >
> > > Please note that the aforementioned points represent only some of our
> > motivations and vision for this donation. Specific future operations need
> > to be further discussed in this thread. For example, the sub-project name
> > after the donation; we hope to name it Flink-CDC
> > > aiming to streaming data intergration through Apache Flink,
> > > following the naming convention of Flink-ML; And this project is
> managed
> > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink
> > Committer. The remaining 4 maintainers are also highly active
> contributors
> > to the Flink community, donating this project to the Flink community
> > implies that their permissions might be reduced. Therefore, we may need
> to
> > bring up this topic for further discussion within the Flink PMC.
> > Additionally, we need to discuss how to migrate existing users and
> > documents. We have a user group of nearly 10,000 people and a
> multi-version
> > documentation site need to migrate. We also need to plan for the
> migration
> > of CI/CD processes and other specifics.
> > >
> > >
> > >
> > > While there are many intricate details that require implementation, we
> > are committed to progressing and finalizing this donation process.
> > >
> > >
> > >
> > > Despite being Flink’s most active ecological project (as evaluated by
> > GitHub metrics), it also boasts a significant user base. However, I
> believe
> > it's essential to commence discussions on future operations only after
> the
> > community reaches a consensus on whether they desire this donation.
> > >
> > >
> > > Really looking forward to hear what you think!
> > >
> > >
> > >
> > > Best,
> > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >
> > > [1] https://github.com/ververica/flink-cdc-connectors
> > > [2]
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > [3] https://debezium.io
> > > [4]
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > >
> >
>


-- 
Best regards,
Sergey

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Feng Jin <ji...@gmail.com>.
This is incredibly exciting news, a big +1 for this.

Thank you for the fantastic work on Flink CDC. We have created thousands of
real-time integration jobs using Flink CDC connectors.


Best,
Feng

On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <go...@apache.org>
wrote:

> It's very exciting to hear the news.
> +1 for adding CDC Connectors  to Apache Flink !
>
>
> Best,
> Zhongqiang
>
> Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
>
> > Dear Flink devs,
> >
> >
> > As you may have heard, we at Alibaba (Ververica) are planning to donate
> CDC Connectors for the Apache Flink project
> > *[1]* to the Apache Flink community.
> >
> >
> >
> > CDC Connectors for Apache Flink comprise a collection of source
> connectors designed specifically for Apache Flink. These connectors
> > *[2]*
> >  enable the ingestion of changes from various databases using Change
> Data Capture (CDC), most of these CDC connectors are powered by Debezium
> > *[3]*
> > . They support both the DataStream API and the Table/SQL API,
> facilitating the reading of database snapshots and continuous reading of
> transaction logs with exactly-once processing, even in the event of
> failures.
> >
> >
> >
> > Additionally, in the latest version 3.0, we have introduced many
> long-awaited features. Starting from CDC version 3.0, we've built a
> Streaming ELT Framework available for streaming data integration. This
> framework allows users to write their data synchronization logic in a
> simple YAML file, which will automatically be translated into a Flink
> DataStreaming job. It emphasizes optimizing the task submission process and
> offers advanced functionalities such as whole database synchronization,
> merging sharded tables, and schema evolution
> > *[4]*.
> >
> >
> >
> >
> > I believe this initiative is a perfect match for both sides. For the
> Flink community, it presents an opportunity to enhance Flink's competitive
> advantage in streaming data integration, promoting the healthy growth and
> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> becoming a sub-project of Apache Flink means being part of a neutral
> open-source community, which can attract a more diverse pool of
> contributors.
> >
> >
> > Please note that the aforementioned points represent only some of our
> motivations and vision for this donation. Specific future operations need
> to be further discussed in this thread. For example, the sub-project name
> after the donation; we hope to name it Flink-CDC
> > aiming to streaming data intergration through Apache Flink,
> > following the naming convention of Flink-ML; And this project is managed
> by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink
> Committer. The remaining 4 maintainers are also highly active contributors
> to the Flink community, donating this project to the Flink community
> implies that their permissions might be reduced. Therefore, we may need to
> bring up this topic for further discussion within the Flink PMC.
> Additionally, we need to discuss how to migrate existing users and
> documents. We have a user group of nearly 10,000 people and a multi-version
> documentation site need to migrate. We also need to plan for the migration
> of CI/CD processes and other specifics.
> >
> >
> >
> > While there are many intricate details that require implementation, we
> are committed to progressing and finalizing this donation process.
> >
> >
> >
> > Despite being Flink’s most active ecological project (as evaluated by
> GitHub metrics), it also boasts a significant user base. However, I believe
> it's essential to commence discussions on future operations only after the
> community reaches a consensus on whether they desire this donation.
> >
> >
> > Really looking forward to hear what you think!
> >
> >
> >
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >
> > [1] https://github.com/ververica/flink-cdc-connectors
> > [2]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3] https://debezium.io
> > [4]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by gongzhongqiang <go...@apache.org>.
It's very exciting to hear the news.
+1 for adding CDC Connectors  to Apache Flink !


Best,
Zhongqiang

Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：

> Dear Flink devs,
>
>
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project
> *[1]* to the Apache Flink community.
>
>
>
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors
> *[2]*
>  enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium
> *[3]*
> . They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
>
>
>
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution
> *[4]*.
>
>
>
>
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
>
>
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC
> aiming to streaming data intergration through Apache Flink,
> following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics.
>
>
>
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
>
>
>
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
>
>
> Really looking forward to hear what you think!
>
>
>
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>
> [1] https://github.com/ververica/flink-cdc-connectors
> [2]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Leonard Xu <xb...@gmail.com>.
Hey, Zarzycki

Thanks for your reply and recognization ! 

> As I understand you're thinking with Flink CDC 3.0 to make the project
> broader than CDC, as more "Streaming Integration" framework. This will
> resemble Kafka Connect, what about renaming the project to something like
> "Flink Connect"? :-)
> It would be amazing to have such a tool, "swiss army knife" for data
> integration, based on Flink.

Flink Community has a lot of connectors thus the Flink Connect name may not suitable.
Although we offers end-to-end streaming integration capability , we still want to use the current name 
Flink CDC as we want to do best in database CDC area and users has accepted/known the name well. 
While the project name holds some significance, I believe its true importance lies in its ability to effectively
 address problems and deliver substantial value to the community users.

Best,
Leonard


> 
> 
> On Tue, Dec 12, 2023 at 3:01 PM Lincoln Lee <li...@gmail.com> wrote:
> 
>> +1
>> Thanks Leonard and all CDC contributors for the great work!
>> 
>> Best,
>> Lincoln Lee
>> 
>> 
>> Dian Fu <di...@gmail.com> 于2023年12月12日周二 19:11写道：
>> 
>>> Big +1 for this! Thanks for driving this effort!
>>> 
>>> Regards,
>>> Dian
>>> 
>>> On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu <xb...@gmail.com> wrote:
>>>> 
>>>> Hey all,
>>>> 
>>>> I want to extend my gratitude for the positive feedback we've received,
>>> particularly from voices in other communities. It is an honor for the
>> Flink
>>> CDC community to have received your recognition.
>>>> 
>>>> Moving forward, I will collaborate with the Flink CDC community
>>> maintainers to prepare a FLIP (Flink Improvement Proposal) document that
>>> will outline specific plans and implementation details.
>>>> 
>>>> Thank you for your support, and we look forward to contributing to
>>> Apache Flink community.
>>>> 
>>>> Best,
>>>> Leonard
>>>> 
>>>>> 2023年12月11日 下午4:12，yh z <zh...@gmail.com> 写道：
>>>>> 
>>>>> Excited to hear the news,
>>>>> +1
>>>>> 
>>>>> Best,
>>>>> Yunhong (Swuferhong)
>>>>> 
>>>>> liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：
>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> Best,
>>>>>> Ron
>>>>>> 
>>>>>> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
>>>>>> 
>>>>>>> So cool, Big +1 for this exciting work.
>>>>>>> 
>>>>>>> On 2023/12/07 03:24:59 Leonard Xu wrote:
>>>>>>>> Dear Flink devs,
>>>>>>>> 
>>>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
>>> donate
>>>>>>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>>>>>>> community.
>>>>>>>> 
>>>>>>>> CDC Connectors for Apache Flink comprise a collection of source
>>>>>>> connectors designed specifically for Apache Flink. These
>>> connectors[2]
>>>>>>> enable the ingestion of changes from various databases using Change
>>> Data
>>>>>>> Capture (CDC), most of these CDC connectors are powered by
>>> Debezium[3].
>>>>>>> They support both the DataStream API and the Table/SQL API,
>>> facilitating
>>>>>>> the reading of database snapshots and continuous reading of
>>> transaction
>>>>>>> logs with exactly-once processing, even in the event of failures.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Additionally, in the latest version 3.0, we have introduced many
>>>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
>>>>>>> Streaming ELT Framework available for streaming data integration.
>>> This
>>>>>>> framework allows users to write their data synchronization logic
>> in a
>>>>>>> simple YAML file, which will automatically be translated into a
>> Flink
>>>>>>> DataStreaming job. It emphasizes optimizing the task submission
>>> process
>>>>>> and
>>>>>>> offers advanced functionalities such as whole database
>>> synchronization,
>>>>>>> merging sharded tables, and schema evolution[4].
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I believe this initiative is a perfect match for both sides. For
>> the
>>>>>>> Flink community, it presents an opportunity to enhance Flink's
>>>>>> competitive
>>>>>>> advantage in streaming data integration, promoting the healthy
>>> growth and
>>>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
>>> project,
>>>>>>> becoming a sub-project of Apache Flink means being part of a
>> neutral
>>>>>>> open-source community, which can attract a more diverse pool of
>>>>>>> contributors.
>>>>>>>> 
>>>>>>>> Please note that the aforementioned points represent only some of
>>> our
>>>>>>> motivations and vision for this donation. Specific future
>> operations
>>> need
>>>>>>> to be further discussed in this thread. For example, the
>> sub-project
>>> name
>>>>>>> after the donation; we hope to name it Flink-CDC aiming to
>> streaming
>>> data
>>>>>>> intergration through Apache Flink, following the naming convention
>> of
>>>>>>> Flink-ML; And this project is managed by a total of 8 maintainers,
>>>>>>> including 3 Flink PMC members and 1 Flink Committer. The remaining
>> 4
>>>>>>> maintainers are also highly active contributors to the Flink
>>> community,
>>>>>>> donating this project to the Flink community implies that their
>>>>>> permissions
>>>>>>> might be reduced. Therefore, we may need to bring up this topic for
>>>>>> further
>>>>>>> discussion within the Flink PMC. Additionally, we need to discuss
>>> how to
>>>>>>> migrate existing users and documents. We have a user group of
>> nearly
>>>>>> 10,000
>>>>>>> people and a multi-version documentation site need to migrate. We
>>> also
>>>>>> need
>>>>>>> to plan for the migration of CI/CD processes and other specifics.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> While there are many intricate details that require
>> implementation,
>>> we
>>>>>>> are committed to progressing and finalizing this donation process.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Despite being Flink’s most active ecological project (as evaluated
>>> by
>>>>>>> GitHub metrics), it also boasts a significant user base. However, I
>>>>>> believe
>>>>>>> it's essential to commence discussions on future operations only
>>> after
>>>>>> the
>>>>>>> community reaches a consensus on whether they desire this donation.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Really looking forward to hear what you think!
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Leonard (on behalf of the Flink CDC Connectors project
>> maintainers)
>>>>>>>> 
>>>>>>>> [1] https://github.com/ververica/flink-cdc-connectors
>>>>>>>> [2]
>>>>>>> 
>>>>>> 
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>>>>>>>> [3] https://debezium.io
>>>>>>>> [4]
>>>>>>> 
>>>>>> 
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Krzysztof Zarzycki <k....@gmail.com>.
Hello, I'm super excited that Flink CDC will join the Flink project! It
will be so much easier to bring it into Flink-based projects we run.

As I understand you're thinking with Flink CDC 3.0 to make the project
broader than CDC, as more "Streaming Integration" framework. This will
resemble Kafka Connect, what about renaming the project to something like
"Flink Connect"? :-)
It would be amazing to have such a tool, "swiss army knife" for data
integration, based on Flink.

Best
Krzysztof



On Tue, Dec 12, 2023 at 3:01 PM Lincoln Lee <li...@gmail.com> wrote:

> +1
> Thanks Leonard and all CDC contributors for the great work!
>
> Best,
> Lincoln Lee
>
>
> Dian Fu <di...@gmail.com> 于2023年12月12日周二 19:11写道：
>
> > Big +1 for this! Thanks for driving this effort!
> >
> > Regards,
> > Dian
> >
> > On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu <xb...@gmail.com> wrote:
> > >
> > > Hey all,
> > >
> > > I want to extend my gratitude for the positive feedback we've received,
> > particularly from voices in other communities. It is an honor for the
> Flink
> > CDC community to have received your recognition.
> > >
> > > Moving forward, I will collaborate with the Flink CDC community
> > maintainers to prepare a FLIP (Flink Improvement Proposal) document that
> > will outline specific plans and implementation details.
> > >
> > > Thank you for your support, and we look forward to contributing to
> > Apache Flink community.
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2023年12月11日 下午4:12，yh z <zh...@gmail.com> 写道：
> > > >
> > > > Excited to hear the news,
> > > > +1
> > > >
> > > > Best,
> > > > Yunhong (Swuferhong)
> > > >
> > > > liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：
> > > >
> > > >> +1
> > > >>
> > > >> Best,
> > > >> Ron
> > > >>
> > > >> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
> > > >>
> > > >>> So cool, Big +1 for this exciting work.
> > > >>>
> > > >>> On 2023/12/07 03:24:59 Leonard Xu wrote:
> > > >>>> Dear Flink devs,
> > > >>>>
> > > >>>> As you may have heard, we at Alibaba (Ververica) are planning to
> > donate
> > > >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > > >>> community.
> > > >>>>
> > > >>>> CDC Connectors for Apache Flink comprise a collection of source
> > > >>> connectors designed specifically for Apache Flink. These
> > connectors[2]
> > > >>> enable the ingestion of changes from various databases using Change
> > Data
> > > >>> Capture (CDC), most of these CDC connectors are powered by
> > Debezium[3].
> > > >>> They support both the DataStream API and the Table/SQL API,
> > facilitating
> > > >>> the reading of database snapshots and continuous reading of
> > transaction
> > > >>> logs with exactly-once processing, even in the event of failures.
> > > >>>>
> > > >>>>
> > > >>>> Additionally, in the latest version 3.0, we have introduced many
> > > >>> long-awaited features. Starting from CDC version 3.0, we've built a
> > > >>> Streaming ELT Framework available for streaming data integration.
> > This
> > > >>> framework allows users to write their data synchronization logic
> in a
> > > >>> simple YAML file, which will automatically be translated into a
> Flink
> > > >>> DataStreaming job. It emphasizes optimizing the task submission
> > process
> > > >> and
> > > >>> offers advanced functionalities such as whole database
> > synchronization,
> > > >>> merging sharded tables, and schema evolution[4].
> > > >>>>
> > > >>>>
> > > >>>> I believe this initiative is a perfect match for both sides. For
> the
> > > >>> Flink community, it presents an opportunity to enhance Flink's
> > > >> competitive
> > > >>> advantage in streaming data integration, promoting the healthy
> > growth and
> > > >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > project,
> > > >>> becoming a sub-project of Apache Flink means being part of a
> neutral
> > > >>> open-source community, which can attract a more diverse pool of
> > > >>> contributors.
> > > >>>>
> > > >>>> Please note that the aforementioned points represent only some of
> > our
> > > >>> motivations and vision for this donation. Specific future
> operations
> > need
> > > >>> to be further discussed in this thread. For example, the
> sub-project
> > name
> > > >>> after the donation; we hope to name it Flink-CDC aiming to
> streaming
> > data
> > > >>> intergration through Apache Flink, following the naming convention
> of
> > > >>> Flink-ML; And this project is managed by a total of 8 maintainers,
> > > >>> including 3 Flink PMC members and 1 Flink Committer. The remaining
> 4
> > > >>> maintainers are also highly active contributors to the Flink
> > community,
> > > >>> donating this project to the Flink community implies that their
> > > >> permissions
> > > >>> might be reduced. Therefore, we may need to bring up this topic for
> > > >> further
> > > >>> discussion within the Flink PMC. Additionally, we need to discuss
> > how to
> > > >>> migrate existing users and documents. We have a user group of
> nearly
> > > >> 10,000
> > > >>> people and a multi-version documentation site need to migrate. We
> > also
> > > >> need
> > > >>> to plan for the migration of CI/CD processes and other specifics.
> > > >>>>
> > > >>>>
> > > >>>> While there are many intricate details that require
> implementation,
> > we
> > > >>> are committed to progressing and finalizing this donation process.
> > > >>>>
> > > >>>>
> > > >>>> Despite being Flink’s most active ecological project (as evaluated
> > by
> > > >>> GitHub metrics), it also boasts a significant user base. However, I
> > > >> believe
> > > >>> it's essential to commence discussions on future operations only
> > after
> > > >> the
> > > >>> community reaches a consensus on whether they desire this donation.
> > > >>>>
> > > >>>>
> > > >>>> Really looking forward to hear what you think!
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>> Leonard (on behalf of the Flink CDC Connectors project
> maintainers)
> > > >>>>
> > > >>>> [1] https://github.com/ververica/flink-cdc-connectors
> > > >>>> [2]
> > > >>>
> > > >>
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > >>>> [3] https://debezium.io
> > > >>>> [4]
> > > >>>
> > > >>
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > >>>
> > > >>
> > >
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Lincoln Lee <li...@gmail.com>.
+1
Thanks Leonard and all CDC contributors for the great work!

Best,
Lincoln Lee


Dian Fu <di...@gmail.com> 于2023年12月12日周二 19:11写道：

> Big +1 for this! Thanks for driving this effort!
>
> Regards,
> Dian
>
> On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> > Hey all,
> >
> > I want to extend my gratitude for the positive feedback we've received,
> particularly from voices in other communities. It is an honor for the Flink
> CDC community to have received your recognition.
> >
> > Moving forward, I will collaborate with the Flink CDC community
> maintainers to prepare a FLIP (Flink Improvement Proposal) document that
> will outline specific plans and implementation details.
> >
> > Thank you for your support, and we look forward to contributing to
> Apache Flink community.
> >
> > Best,
> > Leonard
> >
> > > 2023年12月11日 下午4:12，yh z <zh...@gmail.com> 写道：
> > >
> > > Excited to hear the news,
> > > +1
> > >
> > > Best,
> > > Yunhong (Swuferhong)
> > >
> > > liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：
> > >
> > >> +1
> > >>
> > >> Best,
> > >> Ron
> > >>
> > >> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
> > >>
> > >>> So cool, Big +1 for this exciting work.
> > >>>
> > >>> On 2023/12/07 03:24:59 Leonard Xu wrote:
> > >>>> Dear Flink devs,
> > >>>>
> > >>>> As you may have heard, we at Alibaba (Ververica) are planning to
> donate
> > >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > >>> community.
> > >>>>
> > >>>> CDC Connectors for Apache Flink comprise a collection of source
> > >>> connectors designed specifically for Apache Flink. These
> connectors[2]
> > >>> enable the ingestion of changes from various databases using Change
> Data
> > >>> Capture (CDC), most of these CDC connectors are powered by
> Debezium[3].
> > >>> They support both the DataStream API and the Table/SQL API,
> facilitating
> > >>> the reading of database snapshots and continuous reading of
> transaction
> > >>> logs with exactly-once processing, even in the event of failures.
> > >>>>
> > >>>>
> > >>>> Additionally, in the latest version 3.0, we have introduced many
> > >>> long-awaited features. Starting from CDC version 3.0, we've built a
> > >>> Streaming ELT Framework available for streaming data integration.
> This
> > >>> framework allows users to write their data synchronization logic in a
> > >>> simple YAML file, which will automatically be translated into a Flink
> > >>> DataStreaming job. It emphasizes optimizing the task submission
> process
> > >> and
> > >>> offers advanced functionalities such as whole database
> synchronization,
> > >>> merging sharded tables, and schema evolution[4].
> > >>>>
> > >>>>
> > >>>> I believe this initiative is a perfect match for both sides. For the
> > >>> Flink community, it presents an opportunity to enhance Flink's
> > >> competitive
> > >>> advantage in streaming data integration, promoting the healthy
> growth and
> > >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> project,
> > >>> becoming a sub-project of Apache Flink means being part of a neutral
> > >>> open-source community, which can attract a more diverse pool of
> > >>> contributors.
> > >>>>
> > >>>> Please note that the aforementioned points represent only some of
> our
> > >>> motivations and vision for this donation. Specific future operations
> need
> > >>> to be further discussed in this thread. For example, the sub-project
> name
> > >>> after the donation; we hope to name it Flink-CDC aiming to streaming
> data
> > >>> intergration through Apache Flink, following the naming convention of
> > >>> Flink-ML; And this project is managed by a total of 8 maintainers,
> > >>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> > >>> maintainers are also highly active contributors to the Flink
> community,
> > >>> donating this project to the Flink community implies that their
> > >> permissions
> > >>> might be reduced. Therefore, we may need to bring up this topic for
> > >> further
> > >>> discussion within the Flink PMC. Additionally, we need to discuss
> how to
> > >>> migrate existing users and documents. We have a user group of nearly
> > >> 10,000
> > >>> people and a multi-version documentation site need to migrate. We
> also
> > >> need
> > >>> to plan for the migration of CI/CD processes and other specifics.
> > >>>>
> > >>>>
> > >>>> While there are many intricate details that require implementation,
> we
> > >>> are committed to progressing and finalizing this donation process.
> > >>>>
> > >>>>
> > >>>> Despite being Flink’s most active ecological project (as evaluated
> by
> > >>> GitHub metrics), it also boasts a significant user base. However, I
> > >> believe
> > >>> it's essential to commence discussions on future operations only
> after
> > >> the
> > >>> community reaches a consensus on whether they desire this donation.
> > >>>>
> > >>>>
> > >>>> Really looking forward to hear what you think!
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >>>>
> > >>>> [1] https://github.com/ververica/flink-cdc-connectors
> > >>>> [2]
> > >>>
> > >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > >>>> [3] https://debezium.io
> > >>>> [4]
> > >>>
> > >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > >>>
> > >>
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Dian Fu <di...@gmail.com>.
Big +1 for this! Thanks for driving this effort!

Regards,
Dian

On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu <xb...@gmail.com> wrote:
>
> Hey all,
>
> I want to extend my gratitude for the positive feedback we've received, particularly from voices in other communities. It is an honor for the Flink CDC community to have received your recognition.
>
> Moving forward, I will collaborate with the Flink CDC community maintainers to prepare a FLIP (Flink Improvement Proposal) document that will outline specific plans and implementation details.
>
> Thank you for your support, and we look forward to contributing to Apache Flink community.
>
> Best,
> Leonard
>
> > 2023年12月11日 下午4:12，yh z <zh...@gmail.com> 写道：
> >
> > Excited to hear the news,
> > +1
> >
> > Best,
> > Yunhong (Swuferhong)
> >
> > liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：
> >
> >> +1
> >>
> >> Best,
> >> Ron
> >>
> >> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
> >>
> >>> So cool, Big +1 for this exciting work.
> >>>
> >>> On 2023/12/07 03:24:59 Leonard Xu wrote:
> >>>> Dear Flink devs,
> >>>>
> >>>> As you may have heard, we at Alibaba (Ververica) are planning to donate
> >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> >>> community.
> >>>>
> >>>> CDC Connectors for Apache Flink comprise a collection of source
> >>> connectors designed specifically for Apache Flink. These connectors[2]
> >>> enable the ingestion of changes from various databases using Change Data
> >>> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> >>> They support both the DataStream API and the Table/SQL API, facilitating
> >>> the reading of database snapshots and continuous reading of transaction
> >>> logs with exactly-once processing, even in the event of failures.
> >>>>
> >>>>
> >>>> Additionally, in the latest version 3.0, we have introduced many
> >>> long-awaited features. Starting from CDC version 3.0, we've built a
> >>> Streaming ELT Framework available for streaming data integration. This
> >>> framework allows users to write their data synchronization logic in a
> >>> simple YAML file, which will automatically be translated into a Flink
> >>> DataStreaming job. It emphasizes optimizing the task submission process
> >> and
> >>> offers advanced functionalities such as whole database synchronization,
> >>> merging sharded tables, and schema evolution[4].
> >>>>
> >>>>
> >>>> I believe this initiative is a perfect match for both sides. For the
> >>> Flink community, it presents an opportunity to enhance Flink's
> >> competitive
> >>> advantage in streaming data integration, promoting the healthy growth and
> >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> >>> becoming a sub-project of Apache Flink means being part of a neutral
> >>> open-source community, which can attract a more diverse pool of
> >>> contributors.
> >>>>
> >>>> Please note that the aforementioned points represent only some of our
> >>> motivations and vision for this donation. Specific future operations need
> >>> to be further discussed in this thread. For example, the sub-project name
> >>> after the donation; we hope to name it Flink-CDC aiming to streaming data
> >>> intergration through Apache Flink, following the naming convention of
> >>> Flink-ML; And this project is managed by a total of 8 maintainers,
> >>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> >>> maintainers are also highly active contributors to the Flink community,
> >>> donating this project to the Flink community implies that their
> >> permissions
> >>> might be reduced. Therefore, we may need to bring up this topic for
> >> further
> >>> discussion within the Flink PMC. Additionally, we need to discuss how to
> >>> migrate existing users and documents. We have a user group of nearly
> >> 10,000
> >>> people and a multi-version documentation site need to migrate. We also
> >> need
> >>> to plan for the migration of CI/CD processes and other specifics.
> >>>>
> >>>>
> >>>> While there are many intricate details that require implementation, we
> >>> are committed to progressing and finalizing this donation process.
> >>>>
> >>>>
> >>>> Despite being Flink’s most active ecological project (as evaluated by
> >>> GitHub metrics), it also boasts a significant user base. However, I
> >> believe
> >>> it's essential to commence discussions on future operations only after
> >> the
> >>> community reaches a consensus on whether they desire this donation.
> >>>>
> >>>>
> >>>> Really looking forward to hear what you think!
> >>>>
> >>>>
> >>>> Best,
> >>>> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >>>>
> >>>> [1] https://github.com/ververica/flink-cdc-connectors
> >>>> [2]
> >>>
> >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> >>>> [3] https://debezium.io
> >>>> [4]
> >>>
> >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >>>
> >>
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Leonard Xu <xb...@gmail.com>.
Hey all,

I want to extend my gratitude for the positive feedback we've received, particularly from voices in other communities. It is an honor for the Flink CDC community to have received your recognition.

Moving forward, I will collaborate with the Flink CDC community maintainers to prepare a FLIP (Flink Improvement Proposal) document that will outline specific plans and implementation details. 

Thank you for your support, and we look forward to contributing to Apache Flink community.

Best,
Leonard   

> 2023年12月11日 下午4:12，yh z <zh...@gmail.com> 写道：
> 
> Excited to hear the news,
> +1
> 
> Best,
> Yunhong (Swuferhong)
> 
> liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：
> 
>> +1
>> 
>> Best,
>> Ron
>> 
>> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
>> 
>>> So cool, Big +1 for this exciting work.
>>> 
>>> On 2023/12/07 03:24:59 Leonard Xu wrote:
>>>> Dear Flink devs,
>>>> 
>>>> As you may have heard, we at Alibaba (Ververica) are planning to donate
>>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>>> community.
>>>> 
>>>> CDC Connectors for Apache Flink comprise a collection of source
>>> connectors designed specifically for Apache Flink. These connectors[2]
>>> enable the ingestion of changes from various databases using Change Data
>>> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
>>> They support both the DataStream API and the Table/SQL API, facilitating
>>> the reading of database snapshots and continuous reading of transaction
>>> logs with exactly-once processing, even in the event of failures.
>>>> 
>>>> 
>>>> Additionally, in the latest version 3.0, we have introduced many
>>> long-awaited features. Starting from CDC version 3.0, we've built a
>>> Streaming ELT Framework available for streaming data integration. This
>>> framework allows users to write their data synchronization logic in a
>>> simple YAML file, which will automatically be translated into a Flink
>>> DataStreaming job. It emphasizes optimizing the task submission process
>> and
>>> offers advanced functionalities such as whole database synchronization,
>>> merging sharded tables, and schema evolution[4].
>>>> 
>>>> 
>>>> I believe this initiative is a perfect match for both sides. For the
>>> Flink community, it presents an opportunity to enhance Flink's
>> competitive
>>> advantage in streaming data integration, promoting the healthy growth and
>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
>>> becoming a sub-project of Apache Flink means being part of a neutral
>>> open-source community, which can attract a more diverse pool of
>>> contributors.
>>>> 
>>>> Please note that the aforementioned points represent only some of our
>>> motivations and vision for this donation. Specific future operations need
>>> to be further discussed in this thread. For example, the sub-project name
>>> after the donation; we hope to name it Flink-CDC aiming to streaming data
>>> intergration through Apache Flink, following the naming convention of
>>> Flink-ML; And this project is managed by a total of 8 maintainers,
>>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
>>> maintainers are also highly active contributors to the Flink community,
>>> donating this project to the Flink community implies that their
>> permissions
>>> might be reduced. Therefore, we may need to bring up this topic for
>> further
>>> discussion within the Flink PMC. Additionally, we need to discuss how to
>>> migrate existing users and documents. We have a user group of nearly
>> 10,000
>>> people and a multi-version documentation site need to migrate. We also
>> need
>>> to plan for the migration of CI/CD processes and other specifics.
>>>> 
>>>> 
>>>> While there are many intricate details that require implementation, we
>>> are committed to progressing and finalizing this donation process.
>>>> 
>>>> 
>>>> Despite being Flink’s most active ecological project (as evaluated by
>>> GitHub metrics), it also boasts a significant user base. However, I
>> believe
>>> it's essential to commence discussions on future operations only after
>> the
>>> community reaches a consensus on whether they desire this donation.
>>>> 
>>>> 
>>>> Really looking forward to hear what you think!
>>>> 
>>>> 
>>>> Best,
>>>> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>>>> 
>>>> [1] https://github.com/ververica/flink-cdc-connectors
>>>> [2]
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>>>> [3] https://debezium.io
>>>> [4]
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>> 
>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by yh z <zh...@gmail.com>.
Excited to hear the news,
+1

Best,
Yunhong (Swuferhong)

liu ron <ro...@gmail.com> 于2023年12月11日周一 14:05写道：

> +1
>
> Best,
> Ron
>
> Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：
>
> > So cool, Big +1 for this exciting work.
> >
> > On 2023/12/07 03:24:59 Leonard Xu wrote:
> > > Dear Flink devs,
> > >
> > > As you may have heard, we at Alibaba (Ververica) are planning to donate
> > CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > community.
> > >
> > > CDC Connectors for Apache Flink comprise a collection of source
> > connectors designed specifically for Apache Flink. These connectors[2]
> > enable the ingestion of changes from various databases using Change Data
> > Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> > They support both the DataStream API and the Table/SQL API, facilitating
> > the reading of database snapshots and continuous reading of transaction
> > logs with exactly-once processing, even in the event of failures.
> > >
> > >
> > > Additionally, in the latest version 3.0, we have introduced many
> > long-awaited features. Starting from CDC version 3.0, we've built a
> > Streaming ELT Framework available for streaming data integration. This
> > framework allows users to write their data synchronization logic in a
> > simple YAML file, which will automatically be translated into a Flink
> > DataStreaming job. It emphasizes optimizing the task submission process
> and
> > offers advanced functionalities such as whole database synchronization,
> > merging sharded tables, and schema evolution[4].
> > >
> > >
> > > I believe this initiative is a perfect match for both sides. For the
> > Flink community, it presents an opportunity to enhance Flink's
> competitive
> > advantage in streaming data integration, promoting the healthy growth and
> > prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> > becoming a sub-project of Apache Flink means being part of a neutral
> > open-source community, which can attract a more diverse pool of
> > contributors.
> > >
> > > Please note that the aforementioned points represent only some of our
> > motivations and vision for this donation. Specific future operations need
> > to be further discussed in this thread. For example, the sub-project name
> > after the donation; we hope to name it Flink-CDC aiming to streaming data
> > intergration through Apache Flink, following the naming convention of
> > Flink-ML; And this project is managed by a total of 8 maintainers,
> > including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> > maintainers are also highly active contributors to the Flink community,
> > donating this project to the Flink community implies that their
> permissions
> > might be reduced. Therefore, we may need to bring up this topic for
> further
> > discussion within the Flink PMC. Additionally, we need to discuss how to
> > migrate existing users and documents. We have a user group of nearly
> 10,000
> > people and a multi-version documentation site need to migrate. We also
> need
> > to plan for the migration of CI/CD processes and other specifics.
> > >
> > >
> > > While there are many intricate details that require implementation, we
> > are committed to progressing and finalizing this donation process.
> > >
> > >
> > > Despite being Flink’s most active ecological project (as evaluated by
> > GitHub metrics), it also boasts a significant user base. However, I
> believe
> > it's essential to commence discussions on future operations only after
> the
> > community reaches a consensus on whether they desire this donation.
> > >
> > >
> > > Really looking forward to hear what you think!
> > >
> > >
> > > Best,
> > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >
> > > [1] https://github.com/ververica/flink-cdc-connectors
> > > [2]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > [3] https://debezium.io
> > > [4]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by liu ron <ro...@gmail.com>.
+1

Best,
Ron

Yunqing Mo <yu...@apache.org> 于2023年12月11日周一 12:01写道：

> So cool, Big +1 for this exciting work.
>
> On 2023/12/07 03:24:59 Leonard Xu wrote:
> > Dear Flink devs,
> >
> > As you may have heard, we at Alibaba (Ververica) are planning to donate
> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> community.
> >
> > CDC Connectors for Apache Flink comprise a collection of source
> connectors designed specifically for Apache Flink. These connectors[2]
> enable the ingestion of changes from various databases using Change Data
> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> They support both the DataStream API and the Table/SQL API, facilitating
> the reading of database snapshots and continuous reading of transaction
> logs with exactly-once processing, even in the event of failures.
> >
> >
> > Additionally, in the latest version 3.0, we have introduced many
> long-awaited features. Starting from CDC version 3.0, we've built a
> Streaming ELT Framework available for streaming data integration. This
> framework allows users to write their data synchronization logic in a
> simple YAML file, which will automatically be translated into a Flink
> DataStreaming job. It emphasizes optimizing the task submission process and
> offers advanced functionalities such as whole database synchronization,
> merging sharded tables, and schema evolution[4].
> >
> >
> > I believe this initiative is a perfect match for both sides. For the
> Flink community, it presents an opportunity to enhance Flink's competitive
> advantage in streaming data integration, promoting the healthy growth and
> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> becoming a sub-project of Apache Flink means being part of a neutral
> open-source community, which can attract a more diverse pool of
> contributors.
> >
> > Please note that the aforementioned points represent only some of our
> motivations and vision for this donation. Specific future operations need
> to be further discussed in this thread. For example, the sub-project name
> after the donation; we hope to name it Flink-CDC aiming to streaming data
> intergration through Apache Flink, following the naming convention of
> Flink-ML; And this project is managed by a total of 8 maintainers,
> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> maintainers are also highly active contributors to the Flink community,
> donating this project to the Flink community implies that their permissions
> might be reduced. Therefore, we may need to bring up this topic for further
> discussion within the Flink PMC. Additionally, we need to discuss how to
> migrate existing users and documents. We have a user group of nearly 10,000
> people and a multi-version documentation site need to migrate. We also need
> to plan for the migration of CI/CD processes and other specifics.
> >
> >
> > While there are many intricate details that require implementation, we
> are committed to progressing and finalizing this donation process.
> >
> >
> > Despite being Flink’s most active ecological project (as evaluated by
> GitHub metrics), it also boasts a significant user base. However, I believe
> it's essential to commence discussions on future operations only after the
> community reaches a consensus on whether they desire this donation.
> >
> >
> > Really looking forward to hear what you think!
> >
> >
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >
> > [1] https://github.com/ververica/flink-cdc-connectors
> > [2]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3] https://debezium.io
> > [4]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Yunqing Mo <yu...@apache.org>.
So cool, Big +1 for this exciting work.

On 2023/12/07 03:24:59 Leonard Xu wrote:
> Dear Flink devs,
> 
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> 
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> 
> 
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> 
> 
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> 
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 
> 
> 
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> 
> 
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> 
> 
> Really looking forward to hear what you think! 
> 
> 
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> 
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Shengkai Fang <fs...@gmail.com>.
Thanks for all Flink CDC maintainers great work! Big +1.

Best,
Shengkai

tison <wa...@gmail.com> 于2023年12月7日周四 12:01写道：

> This is very cool! +1 from my side.
>
> Best,
> tison.
>
> Benchao Li <li...@apache.org> 于2023年12月7日周四 11:56写道：
> >
> > Thank you, Leonard and all the Flink CDC maintainers.
> >
> > Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've
> > already taken them as a whole project.
> >
> > Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> > >
> > > Dear Flink devs,
> > >
> > > As you may have heard, we at Alibaba (Ververica) are planning to
> donate CDC Connectors for the Apache Flink project[1] to the Apache Flink
> community.
> > >
> > > CDC Connectors for Apache Flink comprise a collection of source
> connectors designed specifically for Apache Flink. These connectors[2]
> enable the ingestion of changes from various databases using Change Data
> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> They support both the DataStream API and the Table/SQL API, facilitating
> the reading of database snapshots and continuous reading of transaction
> logs with exactly-once processing, even in the event of failures.
> > >
> > >
> > > Additionally, in the latest version 3.0, we have introduced many
> long-awaited features. Starting from CDC version 3.0, we've built a
> Streaming ELT Framework available for streaming data integration. This
> framework allows users to write their data synchronization logic in a
> simple YAML file, which will automatically be translated into a Flink
> DataStreaming job. It emphasizes optimizing the task submission process and
> offers advanced functionalities such as whole database synchronization,
> merging sharded tables, and schema evolution[4].
> > >
> > >
> > > I believe this initiative is a perfect match for both sides. For the
> Flink community, it presents an opportunity to enhance Flink's competitive
> advantage in streaming data integration, promoting the healthy growth and
> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> becoming a sub-project of Apache Flink means being part of a neutral
> open-source community, which can attract a more diverse pool of
> contributors.
> > >
> > > Please note that the aforementioned points represent only some of our
> motivations and vision for this donation. Specific future operations need
> to be further discussed in this thread. For example, the sub-project name
> after the donation; we hope to name it Flink-CDC aiming to streaming data
> intergration through Apache Flink, following the naming convention of
> Flink-ML; And this project is managed by a total of 8 maintainers,
> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> maintainers are also highly active contributors to the Flink community,
> donating this project to the Flink community implies that their permissions
> might be reduced. Therefore, we may need to bring up this topic for further
> discussion within the Flink PMC. Additionally, we need to discuss how to
> migrate existing users and documents. We have a user group of nearly 10,000
> people and a multi-version documentation site need to migrate. We also need
> to plan for the migration of CI/CD processes and other specifics.
> > >
> > >
> > > While there are many intricate details that require implementation, we
> are committed to progressing and finalizing this donation process.
> > >
> > >
> > > Despite being Flink’s most active ecological project (as evaluated by
> GitHub metrics), it also boasts a significant user base. However, I believe
> it's essential to commence discussions on future operations only after the
> community reaches a consensus on whether they desire this donation.
> > >
> > >
> > > Really looking forward to hear what you think!
> > >
> > >
> > > Best,
> > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >
> > > [1] https://github.com/ververica/flink-cdc-connectors
> > > [2]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > [3] https://debezium.io
> > > [4]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by tison <wa...@gmail.com>.
This is very cool! +1 from my side.

Best,
tison.

Benchao Li <li...@apache.org> 于2023年12月7日周四 11:56写道：
>
> Thank you, Leonard and all the Flink CDC maintainers.
>
> Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've
> already taken them as a whole project.
>
> Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
> >
> > Dear Flink devs,
> >
> > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> >
> > CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> >
> >
> > Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> >
> >
> > I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> >
> > Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics.
> >
> >
> > While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> >
> >
> > Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> >
> >
> > Really looking forward to hear what you think!
> >
> >
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >
> > [1] https://github.com/ververica/flink-cdc-connectors
> > [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3] https://debezium.io
> > [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>
>
>
> --
>
> Best,
> Benchao Li

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Benchao Li <li...@apache.org>.
Thank you, Leonard and all the Flink CDC maintainers.

Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've
already taken them as a whole project.

Leonard Xu <xb...@gmail.com> 于2023年12月7日周四 11:25写道：
>
> Dear Flink devs,
>
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
>
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
>
>
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
>
>
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
>
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics.
>
>
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
>
>
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
>
>
> Really looking forward to hear what you think!
>
>
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html



-- 

Best,
Benchao Li

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by MengHui Yu <em...@apache.org>.
Good News. +1

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by The Xia <th...@apache.org>.
+1

On 2023/12/11 02:33:24 Xin Gong wrote:
> 
> good news.
> 
> +1
> 
> Best,
> gongxin
> On 2023/12/07 03:24:59 Leonard Xu wrote:
> > Dear Flink devs,
> > 
> > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> > 
> > CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> > 
> > 
> > Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> > 
> > 
> > I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> > 
> > Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 
> > 
> > 
> > While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> > 
> > 
> > Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> > 
> > 
> > Really looking forward to hear what you think! 
> > 
> > 
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > 
> > [1] https://github.com/ververica/flink-cdc-connectors
> > [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3] https://debezium.io
> > [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Xin Gong <pa...@apache.org>.
good news.

+1

Best,
gongxin
On 2023/12/07 03:24:59 Leonard Xu wrote:
> Dear Flink devs,
> 
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> 
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> 
> 
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> 
> 
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> 
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 
> 
> 
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> 
> 
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> 
> 
> Really looking forward to hear what you think! 
> 
> 
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> 
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by wudi <67...@qq.com.INVALID>.
Awesome, +1

Brs,

di.wu

> 2023年12月7日 上午11:24，Leonard Xu <xb...@gmail.com> 写道：
> 
> Dear Flink devs,
> 
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
> 
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
> 
> 
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
> 
> 
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
> 
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. 
> 
> 
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
> 
> 
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
> 
> 
> Really looking forward to hear what you think! 
> 
> 
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
> 
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Xintong Song <to...@gmail.com>.
A big +1 for this proposal.

Thanks Leonard and the Flink CDC community. While there are many further
details to be discussed, I believe this proposal is aligned with the
long-term interests for both communities.

Best,

Xintong



On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <de...@gmail.com> wrote:

> That's really cool :)
> +1 for the great addition
>
> Bests,
> Samrat
>
> On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com> wrote:
>
> > Wow, Cool, Nice
> >
> > CDC is playing an increasingly important role.
> >
> > +1
> >
> > Best,
> > Jingsong
> >
> > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
> > >
> > > Dear Flink devs,
> > >
> > > As you may have heard, we at Alibaba (Ververica) are planning to donate
> > CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > community.
> > >
> > > CDC Connectors for Apache Flink comprise a collection of source
> > connectors designed specifically for Apache Flink. These connectors[2]
> > enable the ingestion of changes from various databases using Change Data
> > Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> > They support both the DataStream API and the Table/SQL API, facilitating
> > the reading of database snapshots and continuous reading of transaction
> > logs with exactly-once processing, even in the event of failures.
> > >
> > >
> > > Additionally, in the latest version 3.0, we have introduced many
> > long-awaited features. Starting from CDC version 3.0, we've built a
> > Streaming ELT Framework available for streaming data integration. This
> > framework allows users to write their data synchronization logic in a
> > simple YAML file, which will automatically be translated into a Flink
> > DataStreaming job. It emphasizes optimizing the task submission process
> and
> > offers advanced functionalities such as whole database synchronization,
> > merging sharded tables, and schema evolution[4].
> > >
> > >
> > > I believe this initiative is a perfect match for both sides. For the
> > Flink community, it presents an opportunity to enhance Flink's
> competitive
> > advantage in streaming data integration, promoting the healthy growth and
> > prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> > becoming a sub-project of Apache Flink means being part of a neutral
> > open-source community, which can attract a more diverse pool of
> > contributors.
> > >
> > > Please note that the aforementioned points represent only some of our
> > motivations and vision for this donation. Specific future operations need
> > to be further discussed in this thread. For example, the sub-project name
> > after the donation; we hope to name it Flink-CDC aiming to streaming data
> > intergration through Apache Flink, following the naming convention of
> > Flink-ML; And this project is managed by a total of 8 maintainers,
> > including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> > maintainers are also highly active contributors to the Flink community,
> > donating this project to the Flink community implies that their
> permissions
> > might be reduced. Therefore, we may need to bring up this topic for
> further
> > discussion within the Flink PMC. Additionally, we need to discuss how to
> > migrate existing users and documents. We have a user group of nearly
> 10,000
> > people and a multi-version documentation site need to migrate. We also
> need
> > to plan for the migration of CI/CD processes and other specifics.
> > >
> > >
> > > While there are many intricate details that require implementation, we
> > are committed to progressing and finalizing this donation process.
> > >
> > >
> > > Despite being Flink’s most active ecological project (as evaluated by
> > GitHub metrics), it also boasts a significant user base. However, I
> believe
> > it's essential to commence discussions on future operations only after
> the
> > community reaches a consensus on whether they desire this donation.
> > >
> > >
> > > Really looking forward to hear what you think!
> > >
> > >
> > > Best,
> > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >
> > > [1] https://github.com/ververica/flink-cdc-connectors
> > > [2]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > [3] https://debezium.io
> > > [4]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
>

Re:Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Xuyang <xy...@163.com>.
Big +1 for this exciting work.




--

    Best！
    Xuyang





在 2023-12-07 12:06:07，"Samrat Deb" <de...@gmail.com> 写道：
>That's really cool :)
>+1 for the great addition
>
>Bests,
>Samrat
>
>On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com> wrote:
>
>> Wow, Cool, Nice
>>
>> CDC is playing an increasingly important role.
>>
>> +1
>>
>> Best,
>> Jingsong
>>
>> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> > Dear Flink devs,
>> >
>> > As you may have heard, we at Alibaba (Ververica) are planning to donate
>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>> community.
>> >
>> > CDC Connectors for Apache Flink comprise a collection of source
>> connectors designed specifically for Apache Flink. These connectors[2]
>> enable the ingestion of changes from various databases using Change Data
>> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
>> They support both the DataStream API and the Table/SQL API, facilitating
>> the reading of database snapshots and continuous reading of transaction
>> logs with exactly-once processing, even in the event of failures.
>> >
>> >
>> > Additionally, in the latest version 3.0, we have introduced many
>> long-awaited features. Starting from CDC version 3.0, we've built a
>> Streaming ELT Framework available for streaming data integration. This
>> framework allows users to write their data synchronization logic in a
>> simple YAML file, which will automatically be translated into a Flink
>> DataStreaming job. It emphasizes optimizing the task submission process and
>> offers advanced functionalities such as whole database synchronization,
>> merging sharded tables, and schema evolution[4].
>> >
>> >
>> > I believe this initiative is a perfect match for both sides. For the
>> Flink community, it presents an opportunity to enhance Flink's competitive
>> advantage in streaming data integration, promoting the healthy growth and
>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
>> becoming a sub-project of Apache Flink means being part of a neutral
>> open-source community, which can attract a more diverse pool of
>> contributors.
>> >
>> > Please note that the aforementioned points represent only some of our
>> motivations and vision for this donation. Specific future operations need
>> to be further discussed in this thread. For example, the sub-project name
>> after the donation; we hope to name it Flink-CDC aiming to streaming data
>> intergration through Apache Flink, following the naming convention of
>> Flink-ML; And this project is managed by a total of 8 maintainers,
>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
>> maintainers are also highly active contributors to the Flink community,
>> donating this project to the Flink community implies that their permissions
>> might be reduced. Therefore, we may need to bring up this topic for further
>> discussion within the Flink PMC. Additionally, we need to discuss how to
>> migrate existing users and documents. We have a user group of nearly 10,000
>> people and a multi-version documentation site need to migrate. We also need
>> to plan for the migration of CI/CD processes and other specifics.
>> >
>> >
>> > While there are many intricate details that require implementation, we
>> are committed to progressing and finalizing this donation process.
>> >
>> >
>> > Despite being Flink’s most active ecological project (as evaluated by
>> GitHub metrics), it also boasts a significant user base. However, I believe
>> it's essential to commence discussions on future operations only after the
>> community reaches a consensus on whether they desire this donation.
>> >
>> >
>> > Really looking forward to hear what you think!
>> >
>> >
>> > Best,
>> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
>> >
>> > [1] https://github.com/ververica/flink-cdc-connectors
>> > [2]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>> > [3] https://debezium.io
>> > [4]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Jiabao Sun <ji...@xtransfer.cn.INVALID>.
Very excited to hear the news, 
+1 for this proposal.

Best,
Jiabao


> 2023年12月7日 13:08，Rui Fan <19...@gmail.com> 写道：
> 
> It's cool!
> 
> Thanks for the great work of CDC,
> +1 for this donation.
> 
> Best,
> Rui
> 
> On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang <lo...@gmail.com>
> wrote:
> 
>> So cool, Big +1 for this exciting work.
>> 
>> Best
>> Hongshun
>> 
>> On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren <re...@apache.org> wrote:
>> 
>>> Thanks for kicking this off, Leonard!
>>> 
>>> As one of the contributors of the CDC project, I'm truly honored to be
>> part
>>> of the community and so excited to hear the news. CDC project was born
>> from
>>> and developed together with Apache Flink, and we are so proud to be
>>> accepted by more and more users around the world.
>>> 
>>> To put my Flink hat on, I believe having the CDC project as a part of
>>> Apache Flink will broadly expand our ecosystem and the usage scenarios.
>>> Both two projects will benefit from closer cooperation, so +1 from my
>> side.
>>> 
>>> Best,
>>> Qingsheng
>>> 
>>> On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <de...@gmail.com>
>> wrote:
>>> 
>>>> That's really cool :)
>>>> +1 for the great addition
>>>> 
>>>> Bests,
>>>> Samrat
>>>> 
>>>> On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com>
>>> wrote:
>>>> 
>>>>> Wow, Cool, Nice
>>>>> 
>>>>> CDC is playing an increasingly important role.
>>>>> 
>>>>> +1
>>>>> 
>>>>> Best,
>>>>> Jingsong
>>>>> 
>>>>> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
>>>>>> 
>>>>>> Dear Flink devs,
>>>>>> 
>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to
>>> donate
>>>>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>>>>> community.
>>>>>> 
>>>>>> CDC Connectors for Apache Flink comprise a collection of source
>>>>> connectors designed specifically for Apache Flink. These connectors[2]
>>>>> enable the ingestion of changes from various databases using Change
>> Data
>>>>> Capture (CDC), most of these CDC connectors are powered by
>> Debezium[3].
>>>>> They support both the DataStream API and the Table/SQL API,
>> facilitating
>>>>> the reading of database snapshots and continuous reading of
>> transaction
>>>>> logs with exactly-once processing, even in the event of failures.
>>>>>> 
>>>>>> 
>>>>>> Additionally, in the latest version 3.0, we have introduced many
>>>>> long-awaited features. Starting from CDC version 3.0, we've built a
>>>>> Streaming ELT Framework available for streaming data integration. This
>>>>> framework allows users to write their data synchronization logic in a
>>>>> simple YAML file, which will automatically be translated into a Flink
>>>>> DataStreaming job. It emphasizes optimizing the task submission
>> process
>>> and
>>>>> offers advanced functionalities such as whole database
>> synchronization,
>>>>> merging sharded tables, and schema evolution[4].
>>>>>> 
>>>>>> 
>>>>>> I believe this initiative is a perfect match for both sides. For the
>>>>> Flink community, it presents an opportunity to enhance Flink's
>>> competitive
>>>>> advantage in streaming data integration, promoting the healthy growth
>>> and
>>>>> prosperity of the Apache Flink ecosystem. For the CDC Connectors
>>> project,
>>>>> becoming a sub-project of Apache Flink means being part of a neutral
>>>>> open-source community, which can attract a more diverse pool of
>>>>> contributors.
>>>>>> 
>>>>>> Please note that the aforementioned points represent only some of
>> our
>>>>> motivations and vision for this donation. Specific future operations
>>> need
>>>>> to be further discussed in this thread. For example, the sub-project
>>> name
>>>>> after the donation; we hope to name it Flink-CDC aiming to streaming
>>> data
>>>>> intergration through Apache Flink, following the naming convention of
>>>>> Flink-ML; And this project is managed by a total of 8 maintainers,
>>>>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
>>>>> maintainers are also highly active contributors to the Flink
>> community,
>>>>> donating this project to the Flink community implies that their
>>> permissions
>>>>> might be reduced. Therefore, we may need to bring up this topic for
>>> further
>>>>> discussion within the Flink PMC. Additionally, we need to discuss how
>> to
>>>>> migrate existing users and documents. We have a user group of nearly
>>> 10,000
>>>>> people and a multi-version documentation site need to migrate. We also
>>> need
>>>>> to plan for the migration of CI/CD processes and other specifics.
>>>>>> 
>>>>>> 
>>>>>> While there are many intricate details that require implementation,
>> we
>>>>> are committed to progressing and finalizing this donation process.
>>>>>> 
>>>>>> 
>>>>>> Despite being Flink’s most active ecological project (as evaluated
>> by
>>>>> GitHub metrics), it also boasts a significant user base. However, I
>>> believe
>>>>> it's essential to commence discussions on future operations only after
>>> the
>>>>> community reaches a consensus on whether they desire this donation.
>>>>>> 
>>>>>> 
>>>>>> Really looking forward to hear what you think!
>>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>>>>>> 
>>>>>> [1] https://github.com/ververica/flink-cdc-connectors
>>>>>> [2]
>>>>> 
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>>>>>> [3] https://debezium.io
>>>>>> [4]
>>>>> 
>>> 
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>>>> 
>>>> 
>>> 
>>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Rui Fan <19...@gmail.com>.
It's cool!

Thanks for the great work of CDC,
+1 for this donation.

Best,
Rui

On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang <lo...@gmail.com>
wrote:

> So cool, Big +1 for this exciting work.
>
> Best
> Hongshun
>
> On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren <re...@apache.org> wrote:
>
> > Thanks for kicking this off, Leonard!
> >
> > As one of the contributors of the CDC project, I'm truly honored to be
> part
> > of the community and so excited to hear the news. CDC project was born
> from
> > and developed together with Apache Flink, and we are so proud to be
> > accepted by more and more users around the world.
> >
> > To put my Flink hat on, I believe having the CDC project as a part of
> > Apache Flink will broadly expand our ecosystem and the usage scenarios.
> > Both two projects will benefit from closer cooperation, so +1 from my
> side.
> >
> > Best,
> > Qingsheng
> >
> > On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <de...@gmail.com>
> wrote:
> >
> > > That's really cool :)
> > > +1 for the great addition
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com>
> > wrote:
> > >
> > >> Wow, Cool, Nice
> > >>
> > >> CDC is playing an increasingly important role.
> > >>
> > >> +1
> > >>
> > >> Best,
> > >> Jingsong
> > >>
> > >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
> > >> >
> > >> > Dear Flink devs,
> > >> >
> > >> > As you may have heard, we at Alibaba (Ververica) are planning to
> > donate
> > >> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > >> community.
> > >> >
> > >> > CDC Connectors for Apache Flink comprise a collection of source
> > >> connectors designed specifically for Apache Flink. These connectors[2]
> > >> enable the ingestion of changes from various databases using Change
> Data
> > >> Capture (CDC), most of these CDC connectors are powered by
> Debezium[3].
> > >> They support both the DataStream API and the Table/SQL API,
> facilitating
> > >> the reading of database snapshots and continuous reading of
> transaction
> > >> logs with exactly-once processing, even in the event of failures.
> > >> >
> > >> >
> > >> > Additionally, in the latest version 3.0, we have introduced many
> > >> long-awaited features. Starting from CDC version 3.0, we've built a
> > >> Streaming ELT Framework available for streaming data integration. This
> > >> framework allows users to write their data synchronization logic in a
> > >> simple YAML file, which will automatically be translated into a Flink
> > >> DataStreaming job. It emphasizes optimizing the task submission
> process
> > and
> > >> offers advanced functionalities such as whole database
> synchronization,
> > >> merging sharded tables, and schema evolution[4].
> > >> >
> > >> >
> > >> > I believe this initiative is a perfect match for both sides. For the
> > >> Flink community, it presents an opportunity to enhance Flink's
> > competitive
> > >> advantage in streaming data integration, promoting the healthy growth
> > and
> > >> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> > project,
> > >> becoming a sub-project of Apache Flink means being part of a neutral
> > >> open-source community, which can attract a more diverse pool of
> > >> contributors.
> > >> >
> > >> > Please note that the aforementioned points represent only some of
> our
> > >> motivations and vision for this donation. Specific future operations
> > need
> > >> to be further discussed in this thread. For example, the sub-project
> > name
> > >> after the donation; we hope to name it Flink-CDC aiming to streaming
> > data
> > >> intergration through Apache Flink, following the naming convention of
> > >> Flink-ML; And this project is managed by a total of 8 maintainers,
> > >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> > >> maintainers are also highly active contributors to the Flink
> community,
> > >> donating this project to the Flink community implies that their
> > permissions
> > >> might be reduced. Therefore, we may need to bring up this topic for
> > further
> > >> discussion within the Flink PMC. Additionally, we need to discuss how
> to
> > >> migrate existing users and documents. We have a user group of nearly
> > 10,000
> > >> people and a multi-version documentation site need to migrate. We also
> > need
> > >> to plan for the migration of CI/CD processes and other specifics.
> > >> >
> > >> >
> > >> > While there are many intricate details that require implementation,
> we
> > >> are committed to progressing and finalizing this donation process.
> > >> >
> > >> >
> > >> > Despite being Flink’s most active ecological project (as evaluated
> by
> > >> GitHub metrics), it also boasts a significant user base. However, I
> > believe
> > >> it's essential to commence discussions on future operations only after
> > the
> > >> community reaches a consensus on whether they desire this donation.
> > >> >
> > >> >
> > >> > Really looking forward to hear what you think!
> > >> >
> > >> >
> > >> > Best,
> > >> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >> >
> > >> > [1] https://github.com/ververica/flink-cdc-connectors
> > >> > [2]
> > >>
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > >> > [3] https://debezium.io
> > >> > [4]
> > >>
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > >>
> > >
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Hongshun Wang <lo...@gmail.com>.
So cool, Big +1 for this exciting work.

Best
Hongshun

On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren <re...@apache.org> wrote:

> Thanks for kicking this off, Leonard!
>
> As one of the contributors of the CDC project, I'm truly honored to be part
> of the community and so excited to hear the news. CDC project was born from
> and developed together with Apache Flink, and we are so proud to be
> accepted by more and more users around the world.
>
> To put my Flink hat on, I believe having the CDC project as a part of
> Apache Flink will broadly expand our ecosystem and the usage scenarios.
> Both two projects will benefit from closer cooperation, so +1 from my side.
>
> Best,
> Qingsheng
>
> On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <de...@gmail.com> wrote:
>
> > That's really cool :)
> > +1 for the great addition
> >
> > Bests,
> > Samrat
> >
> > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com>
> wrote:
> >
> >> Wow, Cool, Nice
> >>
> >> CDC is playing an increasingly important role.
> >>
> >> +1
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
> >> >
> >> > Dear Flink devs,
> >> >
> >> > As you may have heard, we at Alibaba (Ververica) are planning to
> donate
> >> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> >> community.
> >> >
> >> > CDC Connectors for Apache Flink comprise a collection of source
> >> connectors designed specifically for Apache Flink. These connectors[2]
> >> enable the ingestion of changes from various databases using Change Data
> >> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> >> They support both the DataStream API and the Table/SQL API, facilitating
> >> the reading of database snapshots and continuous reading of transaction
> >> logs with exactly-once processing, even in the event of failures.
> >> >
> >> >
> >> > Additionally, in the latest version 3.0, we have introduced many
> >> long-awaited features. Starting from CDC version 3.0, we've built a
> >> Streaming ELT Framework available for streaming data integration. This
> >> framework allows users to write their data synchronization logic in a
> >> simple YAML file, which will automatically be translated into a Flink
> >> DataStreaming job. It emphasizes optimizing the task submission process
> and
> >> offers advanced functionalities such as whole database synchronization,
> >> merging sharded tables, and schema evolution[4].
> >> >
> >> >
> >> > I believe this initiative is a perfect match for both sides. For the
> >> Flink community, it presents an opportunity to enhance Flink's
> competitive
> >> advantage in streaming data integration, promoting the healthy growth
> and
> >> prosperity of the Apache Flink ecosystem. For the CDC Connectors
> project,
> >> becoming a sub-project of Apache Flink means being part of a neutral
> >> open-source community, which can attract a more diverse pool of
> >> contributors.
> >> >
> >> > Please note that the aforementioned points represent only some of our
> >> motivations and vision for this donation. Specific future operations
> need
> >> to be further discussed in this thread. For example, the sub-project
> name
> >> after the donation; we hope to name it Flink-CDC aiming to streaming
> data
> >> intergration through Apache Flink, following the naming convention of
> >> Flink-ML; And this project is managed by a total of 8 maintainers,
> >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> >> maintainers are also highly active contributors to the Flink community,
> >> donating this project to the Flink community implies that their
> permissions
> >> might be reduced. Therefore, we may need to bring up this topic for
> further
> >> discussion within the Flink PMC. Additionally, we need to discuss how to
> >> migrate existing users and documents. We have a user group of nearly
> 10,000
> >> people and a multi-version documentation site need to migrate. We also
> need
> >> to plan for the migration of CI/CD processes and other specifics.
> >> >
> >> >
> >> > While there are many intricate details that require implementation, we
> >> are committed to progressing and finalizing this donation process.
> >> >
> >> >
> >> > Despite being Flink’s most active ecological project (as evaluated by
> >> GitHub metrics), it also boasts a significant user base. However, I
> believe
> >> it's essential to commence discussions on future operations only after
> the
> >> community reaches a consensus on whether they desire this donation.
> >> >
> >> >
> >> > Really looking forward to hear what you think!
> >> >
> >> >
> >> > Best,
> >> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >> >
> >> > [1] https://github.com/ververica/flink-cdc-connectors
> >> > [2]
> >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> >> > [3] https://debezium.io
> >> > [4]
> >>
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >>
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Qingsheng Ren <re...@apache.org>.
Thanks for kicking this off, Leonard!

As one of the contributors of the CDC project, I'm truly honored to be part
of the community and so excited to hear the news. CDC project was born from
and developed together with Apache Flink, and we are so proud to be
accepted by more and more users around the world.

To put my Flink hat on, I believe having the CDC project as a part of
Apache Flink will broadly expand our ecosystem and the usage scenarios.
Both two projects will benefit from closer cooperation, so +1 from my side.

Best,
Qingsheng

On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <de...@gmail.com> wrote:

> That's really cool :)
> +1 for the great addition
>
> Bests,
> Samrat
>
> On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com> wrote:
>
>> Wow, Cool, Nice
>>
>> CDC is playing an increasingly important role.
>>
>> +1
>>
>> Best,
>> Jingsong
>>
>> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> > Dear Flink devs,
>> >
>> > As you may have heard, we at Alibaba (Ververica) are planning to donate
>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>> community.
>> >
>> > CDC Connectors for Apache Flink comprise a collection of source
>> connectors designed specifically for Apache Flink. These connectors[2]
>> enable the ingestion of changes from various databases using Change Data
>> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
>> They support both the DataStream API and the Table/SQL API, facilitating
>> the reading of database snapshots and continuous reading of transaction
>> logs with exactly-once processing, even in the event of failures.
>> >
>> >
>> > Additionally, in the latest version 3.0, we have introduced many
>> long-awaited features. Starting from CDC version 3.0, we've built a
>> Streaming ELT Framework available for streaming data integration. This
>> framework allows users to write their data synchronization logic in a
>> simple YAML file, which will automatically be translated into a Flink
>> DataStreaming job. It emphasizes optimizing the task submission process and
>> offers advanced functionalities such as whole database synchronization,
>> merging sharded tables, and schema evolution[4].
>> >
>> >
>> > I believe this initiative is a perfect match for both sides. For the
>> Flink community, it presents an opportunity to enhance Flink's competitive
>> advantage in streaming data integration, promoting the healthy growth and
>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
>> becoming a sub-project of Apache Flink means being part of a neutral
>> open-source community, which can attract a more diverse pool of
>> contributors.
>> >
>> > Please note that the aforementioned points represent only some of our
>> motivations and vision for this donation. Specific future operations need
>> to be further discussed in this thread. For example, the sub-project name
>> after the donation; we hope to name it Flink-CDC aiming to streaming data
>> intergration through Apache Flink, following the naming convention of
>> Flink-ML; And this project is managed by a total of 8 maintainers,
>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
>> maintainers are also highly active contributors to the Flink community,
>> donating this project to the Flink community implies that their permissions
>> might be reduced. Therefore, we may need to bring up this topic for further
>> discussion within the Flink PMC. Additionally, we need to discuss how to
>> migrate existing users and documents. We have a user group of nearly 10,000
>> people and a multi-version documentation site need to migrate. We also need
>> to plan for the migration of CI/CD processes and other specifics.
>> >
>> >
>> > While there are many intricate details that require implementation, we
>> are committed to progressing and finalizing this donation process.
>> >
>> >
>> > Despite being Flink’s most active ecological project (as evaluated by
>> GitHub metrics), it also boasts a significant user base. However, I believe
>> it's essential to commence discussions on future operations only after the
>> community reaches a consensus on whether they desire this donation.
>> >
>> >
>> > Really looking forward to hear what you think!
>> >
>> >
>> > Best,
>> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
>> >
>> > [1] https://github.com/ververica/flink-cdc-connectors
>> > [2]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>> > [3] https://debezium.io
>> > [4]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Jark Wu <im...@gmail.com>.
+1 for adding this to Apache Flink!

I think this can further extend the ability of Apache Flink and a lot of
users would be
interested to try this out.

Best,
Jark

On Thu, 7 Dec 2023 at 12:06, Samrat Deb <de...@gmail.com> wrote:

> That's really cool :)
> +1 for the great addition
>
> Bests,
> Samrat
>
> On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com> wrote:
>
>> Wow, Cool, Nice
>>
>> CDC is playing an increasingly important role.
>>
>> +1
>>
>> Best,
>> Jingsong
>>
>> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
>> >
>> > Dear Flink devs,
>> >
>> > As you may have heard, we at Alibaba (Ververica) are planning to donate
>> CDC Connectors for the Apache Flink project[1] to the Apache Flink
>> community.
>> >
>> > CDC Connectors for Apache Flink comprise a collection of source
>> connectors designed specifically for Apache Flink. These connectors[2]
>> enable the ingestion of changes from various databases using Change Data
>> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
>> They support both the DataStream API and the Table/SQL API, facilitating
>> the reading of database snapshots and continuous reading of transaction
>> logs with exactly-once processing, even in the event of failures.
>> >
>> >
>> > Additionally, in the latest version 3.0, we have introduced many
>> long-awaited features. Starting from CDC version 3.0, we've built a
>> Streaming ELT Framework available for streaming data integration. This
>> framework allows users to write their data synchronization logic in a
>> simple YAML file, which will automatically be translated into a Flink
>> DataStreaming job. It emphasizes optimizing the task submission process and
>> offers advanced functionalities such as whole database synchronization,
>> merging sharded tables, and schema evolution[4].
>> >
>> >
>> > I believe this initiative is a perfect match for both sides. For the
>> Flink community, it presents an opportunity to enhance Flink's competitive
>> advantage in streaming data integration, promoting the healthy growth and
>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
>> becoming a sub-project of Apache Flink means being part of a neutral
>> open-source community, which can attract a more diverse pool of
>> contributors.
>> >
>> > Please note that the aforementioned points represent only some of our
>> motivations and vision for this donation. Specific future operations need
>> to be further discussed in this thread. For example, the sub-project name
>> after the donation; we hope to name it Flink-CDC aiming to streaming data
>> intergration through Apache Flink, following the naming convention of
>> Flink-ML; And this project is managed by a total of 8 maintainers,
>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
>> maintainers are also highly active contributors to the Flink community,
>> donating this project to the Flink community implies that their permissions
>> might be reduced. Therefore, we may need to bring up this topic for further
>> discussion within the Flink PMC. Additionally, we need to discuss how to
>> migrate existing users and documents. We have a user group of nearly 10,000
>> people and a multi-version documentation site need to migrate. We also need
>> to plan for the migration of CI/CD processes and other specifics.
>> >
>> >
>> > While there are many intricate details that require implementation, we
>> are committed to progressing and finalizing this donation process.
>> >
>> >
>> > Despite being Flink’s most active ecological project (as evaluated by
>> GitHub metrics), it also boasts a significant user base. However, I believe
>> it's essential to commence discussions on future operations only after the
>> community reaches a consensus on whether they desire this donation.
>> >
>> >
>> > Really looking forward to hear what you think!
>> >
>> >
>> > Best,
>> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
>> >
>> > [1] https://github.com/ververica/flink-cdc-connectors
>> > [2]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
>> > [3] https://debezium.io
>> > [4]
>> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>>
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Samrat Deb <de...@gmail.com>.
That's really cool :)
+1 for the great addition

Bests,
Samrat

On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <ji...@gmail.com> wrote:

> Wow, Cool, Nice
>
> CDC is playing an increasingly important role.
>
> +1
>
> Best,
> Jingsong
>
> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
> >
> > Dear Flink devs,
> >
> > As you may have heard, we at Alibaba (Ververica) are planning to donate
> CDC Connectors for the Apache Flink project[1] to the Apache Flink
> community.
> >
> > CDC Connectors for Apache Flink comprise a collection of source
> connectors designed specifically for Apache Flink. These connectors[2]
> enable the ingestion of changes from various databases using Change Data
> Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> They support both the DataStream API and the Table/SQL API, facilitating
> the reading of database snapshots and continuous reading of transaction
> logs with exactly-once processing, even in the event of failures.
> >
> >
> > Additionally, in the latest version 3.0, we have introduced many
> long-awaited features. Starting from CDC version 3.0, we've built a
> Streaming ELT Framework available for streaming data integration. This
> framework allows users to write their data synchronization logic in a
> simple YAML file, which will automatically be translated into a Flink
> DataStreaming job. It emphasizes optimizing the task submission process and
> offers advanced functionalities such as whole database synchronization,
> merging sharded tables, and schema evolution[4].
> >
> >
> > I believe this initiative is a perfect match for both sides. For the
> Flink community, it presents an opportunity to enhance Flink's competitive
> advantage in streaming data integration, promoting the healthy growth and
> prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> becoming a sub-project of Apache Flink means being part of a neutral
> open-source community, which can attract a more diverse pool of
> contributors.
> >
> > Please note that the aforementioned points represent only some of our
> motivations and vision for this donation. Specific future operations need
> to be further discussed in this thread. For example, the sub-project name
> after the donation; we hope to name it Flink-CDC aiming to streaming data
> intergration through Apache Flink, following the naming convention of
> Flink-ML; And this project is managed by a total of 8 maintainers,
> including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> maintainers are also highly active contributors to the Flink community,
> donating this project to the Flink community implies that their permissions
> might be reduced. Therefore, we may need to bring up this topic for further
> discussion within the Flink PMC. Additionally, we need to discuss how to
> migrate existing users and documents. We have a user group of nearly 10,000
> people and a multi-version documentation site need to migrate. We also need
> to plan for the migration of CI/CD processes and other specifics.
> >
> >
> > While there are many intricate details that require implementation, we
> are committed to progressing and finalizing this donation process.
> >
> >
> > Despite being Flink’s most active ecological project (as evaluated by
> GitHub metrics), it also boasts a significant user base. However, I believe
> it's essential to commence discussions on future operations only after the
> community reaches a consensus on whether they desire this donation.
> >
> >
> > Really looking forward to hear what you think!
> >
> >
> > Best,
> > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> >
> > [1] https://github.com/ververica/flink-cdc-connectors
> > [2]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > [3] https://debezium.io
> > [4]
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Jingsong Li <ji...@gmail.com>.
Wow, Cool, Nice

CDC is playing an increasingly important role.

+1

Best,
Jingsong

On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xb...@gmail.com> wrote:
>
> Dear Flink devs,
>
> As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community.
>
> CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures.
>
>
> Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4].
>
>
> I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors.
>
> Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics.
>
>
> While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process.
>
>
> Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation.
>
>
> Really looking forward to hear what you think!
>
>
> Best,
> Leonard (on behalf of the Flink CDC Connectors project maintainers)
>
> [1] https://github.com/ververica/flink-cdc-connectors
> [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> [3] https://debezium.io
> [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

Posted by Jiafeng Zhang <ji...@apache.org>.
+1
Great, cdc is very important for real-time data integration, and this project is also widely used, I am optimistic about this project

Jiafeng.Zhang / 张家锋
Email: jiafengzheng@apache.org