You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by Zongwen Li <zo...@apache.org> on 2022/10/25 09:18:56 UTC
[DISCUSS] CDC Connector Design
hi, all
The CDC connector is mentioned several times among developers and
users as necessary,
If you want to understand the background of the matter, you can see
https://github.com/apache/incubator-seatunnel/issues/2394
I have completed the preliminary design of the CDC connector, you can
review the design:
https://github.com/apache/incubator-seatunnel/issues/3175
and give me feedback.
The following are the goals of this design:
- Support parallel reading of historical data (Fast synchronization,
billions of large table)
- Support reading incremental data (CDC)
- Support for dynamically adding new tables (Easier to operate and maintain)
- Support multi-table and sharding (Easy Configuration)
--
Best Regards,
Zongwen Li
Re: [DISCUSS] CDC Connector Design
Posted by JUN GAO <ga...@apache.org>.
That's great, I believe that the CDC connector will make SeaTunnel better.
Zongwen Li <zo...@apache.org> 于2022年10月25日周二 17:19写道：
> hi, all
>
> The CDC connector is mentioned several times among developers and
> users as necessary,
> If you want to understand the background of the matter, you can see
> https://github.com/apache/incubator-seatunnel/issues/2394
>
> I have completed the preliminary design of the CDC connector, you can
> review the design:
> https://github.com/apache/incubator-seatunnel/issues/3175
> and give me feedback.
>
> The following are the goals of this design:
> - Support parallel reading of historical data (Fast synchronization,
> billions of large table)
> - Support reading incremental data (CDC)
> - Support for dynamically adding new tables (Easier to operate and
> maintain)
> - Support multi-table and sharding (Easy Configuration)
>
> --
> Best Regards,
>
> Zongwen Li
>
--
Best Regards
------------
EricJoy2048
gaojun2048@gmail.com
Re: [DISCUSS] CDC Connector Design
Posted by Lifeng Nie <ni...@apache.org>.
+1 good idea
Leonard Xu <xb...@gmail.com> 于2022年10月26日周三 11:20写道：
> +1 from my side.
>
> Best,
> Leonard
>
> > 2022年10月26日 上午10:40，hailin0 <wa...@apache.org> 写道：
> >
> > +1 good idea
> >
> > Zongwen Li <zo...@apache.org> 于2022年10月25日周二 17:20写道：
> >
> >> hi, all
> >>
> >> The CDC connector is mentioned several times among developers and
> >> users as necessary,
> >> If you want to understand the background of the matter, you can see
> >> https://github.com/apache/incubator-seatunnel/issues/2394
> >>
> >> I have completed the preliminary design of the CDC connector, you can
> >> review the design:
> >> https://github.com/apache/incubator-seatunnel/issues/3175
> >> and give me feedback.
> >>
> >> The following are the goals of this design:
> >> - Support parallel reading of historical data (Fast synchronization,
> >> billions of large table)
> >> - Support reading incremental data (CDC)
> >> - Support for dynamically adding new tables (Easier to operate and
> >> maintain)
> >> - Support multi-table and sharding (Easy Configuration)
> >>
> >> --
> >> Best Regards,
> >>
> >> Zongwen Li
> >>
>
>
--
Warm Regards,
Lifeng
Re: [DISCUSS] CDC Connector Design
Posted by Leonard Xu <xb...@gmail.com>.
+1 from my side.
Best,
Leonard
> 2022年10月26日 上午10:40，hailin0 <wa...@apache.org> 写道：
>
> +1 good idea
>
> Zongwen Li <zo...@apache.org> 于2022年10月25日周二 17:20写道：
>
>> hi, all
>>
>> The CDC connector is mentioned several times among developers and
>> users as necessary,
>> If you want to understand the background of the matter, you can see
>> https://github.com/apache/incubator-seatunnel/issues/2394
>>
>> I have completed the preliminary design of the CDC connector, you can
>> review the design:
>> https://github.com/apache/incubator-seatunnel/issues/3175
>> and give me feedback.
>>
>> The following are the goals of this design:
>> - Support parallel reading of historical data (Fast synchronization,
>> billions of large table)
>> - Support reading incremental data (CDC)
>> - Support for dynamically adding new tables (Easier to operate and
>> maintain)
>> - Support multi-table and sharding (Easy Configuration)
>>
>> --
>> Best Regards,
>>
>> Zongwen Li
>>
Re: [DISCUSS] CDC Connector Design
Posted by hailin0 <wa...@apache.org>.
+1 good idea
Zongwen Li <zo...@apache.org> 于2022年10月25日周二 17:20写道：
> hi, all
>
> The CDC connector is mentioned several times among developers and
> users as necessary,
> If you want to understand the background of the matter, you can see
> https://github.com/apache/incubator-seatunnel/issues/2394
>
> I have completed the preliminary design of the CDC connector, you can
> review the design:
> https://github.com/apache/incubator-seatunnel/issues/3175
> and give me feedback.
>
> The following are the goals of this design:
> - Support parallel reading of historical data (Fast synchronization,
> billions of large table)
> - Support reading incremental data (CDC)
> - Support for dynamically adding new tables (Easier to operate and
> maintain)
> - Support multi-table and sharding (Easy Configuration)
>
> --
> Best Regards,
>
> Zongwen Li
>