You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by Zongwen Li <zo...@gmail.com> on 2022/05/07 09:36:21 UTC
Re: [DISCUSS] Decoupling connectors from compute engines
The goal of Apache SeaTunnel is different from Apache Beam.
Apache SeaTunnel focuses on source and sink connectors, and develops
features in the field of data integration;
Apache Beam focuses and unifies all the functions of the compute engine,
including operators such as join, connect, map, etc. and it doesn't unify
streaming and batch source.
This improvement proposal is to solve the current problems encountered by
SeaTunnel . If you have better ideas, you can bring them up for discussion.
Best,
Zongwen Li
leo65535 <le...@163.com> 于2022年4月29日周五 16:14写道：
>
>
> Hi @zongwen,
>
>
> I think this is not a good idea, it seems that we will be more and more
> like Apache Beam,
>
>
> Best,
> Leo65535
>
>
> At 2022-04-18 15:10:08, "李宗文" <zo...@gmail.com> wrote:
> >Hi All,
> >In the current implementation of SeaTunnel, the connector is coupled with
> >the computing engine, which results in a connector that needs to be
> >implemented for each engine, and it is difficult to support multiple
> >versions of the engine.
> >
> >Through the questionnaire, it was found that users used multiple versions
> >of Spark and Flink engines, and they also hoped that SeaTunnel would
> >support Change Data Capture (CDC) connectors;
> >
> >Based on the above questions and needs, I created an improvement proposal:
> >https://github.com/apache/incubator-seatunnel/issues/1608
> >Preliminary idea of Source and Sink API:
> > https://github.com/apache/incubator-seatunnel/issues/1701
> > https://github.com/apache/incubator-seatunnel/issues/1704
> >
> >Please discuss away! Zongwen Li
>
Re: [DISCUSS] Decoupling connectors from compute engines
Posted by 范佳 <fa...@qq.com.INVALID>.
I agree with zongwen's point of view, the purpose of making your own API is to help developers develop Connectors suitable for SeaTunnel better and faster. It can be understood that our Connector supports Spark and Flink, not that our Connector depends on Spark and Flink. I think this point is important so that we can do SeaTunnel's own stuff.
> 2022年5月7日 17:36，Zongwen Li <zo...@gmail.com> 写道：
>
> The goal of Apache SeaTunnel is different from Apache Beam.
> Apache SeaTunnel focuses on source and sink connectors, and develops
> features in the field of data integration;
> Apache Beam focuses and unifies all the functions of the compute engine,
> including operators such as join, connect, map, etc. and it doesn't unify
> streaming and batch source.
>
> This improvement proposal is to solve the current problems encountered by
> SeaTunnel . If you have better ideas, you can bring them up for discussion.
>
> Best,
> Zongwen Li
>
> leo65535 <le...@163.com> 于2022年4月29日周五 16:14写道：
>
>>
>>
>> Hi @zongwen,
>>
>>
>> I think this is not a good idea, it seems that we will be more and more
>> like Apache Beam,
>>
>>
>> Best,
>> Leo65535
>>
>>
>> At 2022-04-18 15:10:08, "李宗文" <zo...@gmail.com> wrote:
>>> Hi All,
>>> In the current implementation of SeaTunnel, the connector is coupled with
>>> the computing engine, which results in a connector that needs to be
>>> implemented for each engine, and it is difficult to support multiple
>>> versions of the engine.
>>>
>>> Through the questionnaire, it was found that users used multiple versions
>>> of Spark and Flink engines, and they also hoped that SeaTunnel would
>>> support Change Data Capture (CDC) connectors;
>>>
>>> Based on the above questions and needs, I created an improvement proposal:
>>> https://github.com/apache/incubator-seatunnel/issues/1608
>>> Preliminary idea of Source and Sink API:
>>> https://github.com/apache/incubator-seatunnel/issues/1701
>>> https://github.com/apache/incubator-seatunnel/issues/1704
>>>
>>> Please discuss away! Zongwen Li
>>
>