You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@seatunnel.apache.org by TaoZex <26...@qq.com.INVALID> on 2022/10/19 07:24:02 UTC

RE：[DISCUSS] Add SeaTunnel Transform API(transform-v2) and support spi auto loading

+1


The rapid development of the ST engine also requires transformer-v2, which is a good proposal. When the api is ready, the community can contribute various functional implementations of transform together.




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <wanghailin@apache.org&gt;;
发送时间:&nbsp;2022年10月19日(星期三) 下午2:48
收件人:&nbsp;"dev"<dev@seatunnel.apache.org&gt;;

主题:&nbsp;[DISCUSS] Add SeaTunnel Transform API(transform-v2) and support spi auto loading



Hi developers,

Currently, the transform[1] code is bound to the single engine and cannot
be shared to other engine using.

I propose that we create `transform-v2` module to unify transform
implement, like source and sink, it is decoupled from the engine and can
run on different engines.

Furthermore,&nbsp; we can use the `translation`[2] module to integrate transform
to seatunnel, flink, spark engine execute.

In order to ensure seatunnel's positioning as a data integration platform
and not introduce work beyond the plan, the `transform-v2` will only
support UDF level data conversion, and And unsupported sql
transform(because st-engine unsupported sql parse &amp; analysis).


If you have any suggestions or concerns, you are very welcome to comment in
this email thread or in issue #2678 [3]


[1]
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-transforms
[2]
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-translation
[3] https://github.com/apache/incubator-seatunnel/issues/2678

Re: [DISCUSS] Add SeaTunnel Transform API(transform-v2) and support spi auto loading

Posted by JUN GAO <ga...@apache.org>.

+1, That's a good idea.

TaoZex <26...@qq.com.invalid> 于2022年10月19日周三 15:24写道：

> +1
>
>
> The rapid development of the ST engine also requires transformer-v2, which
> is a good proposal. When the api is ready, the community can contribute
> various functional implementations of transform together.
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
>                                                   "dev"
>                                                                 <
> wanghailin@apache.org&gt;;
> 发送时间:&nbsp;2022年10月19日(星期三) 下午2:48
> 收件人:&nbsp;"dev"<dev@seatunnel.apache.org&gt;;
>
> 主题:&nbsp;[DISCUSS] Add SeaTunnel Transform API(transform-v2) and support
> spi auto loading
>
>
>
> Hi developers,
>
> Currently, the transform[1] code is bound to the single engine and cannot
> be shared to other engine using.
>
> I propose that we create `transform-v2` module to unify transform
> implement, like source and sink, it is decoupled from the engine and can
> run on different engines.
>
> Furthermore,&nbsp; we can use the `translation`[2] module to integrate
> transform
> to seatunnel, flink, spark engine execute.
>
> In order to ensure seatunnel's positioning as a data integration platform
> and not introduce work beyond the plan, the `transform-v2` will only
> support UDF level data conversion, and And unsupported sql
> transform(because st-engine unsupported sql parse &amp; analysis).
>
>
> If you have any suggestions or concerns, you are very welcome to comment in
> this email thread or in issue #2678 [3]
>
>
> [1]
> https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-transforms
> [2]
>
> https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-translation
> [3] https://github.com/apache/incubator-seatunnel/issues/2678



-- 

Best Regards

------------

EricJoy2048
gaojun2048@gmail.com