You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by JUN GAO <ga...@apache.org> on 2023/07/22 02:17:03 UTC

[Proposal] Spark&Flink compatible catalog

*background*

Due to the existence of commercial versions and open source web, it is
necessary to clearly know the data source table and the table structure
processed by each node when defining tasks. So the entire architecture
layer needs to use a catalog approach, and the previous description of
SeaTunnelRowType cannot meet the synchronization requirements.
At present, the Zeta engine has supported the creation of connectors using
the catalog method, but spark/link still uses the SeaTunnelRowType
interface, which requires the connector to implement both the Catalog and
SeaTunnelRowType interfaces in order to execute on different engines. This
will require a lot of work and is meaningless.
To solve this problem, we need spark/link to first support the Catalog
method, so that the new connector can only implement the Catalog interface,
while the old connector can gradually remove the SeaTunnelRowType interface.

-- 

Best Regards

------------

EricJoy2048
gaojun2048@gmail.com

Re: [Proposal] Spark&Flink compatible catalog

Posted by Lucifer Tyrant <ty...@apache.org>.
+1, SeaTunnel the next version will strive to reduce user study costs,
this feature can reduce the difficulty of getting started with
seatunnel.

hailin0 <wa...@apache.org> 于2023年7月22日周六 15:50写道:
>
> +1
>
> Agree with you
>
> TaoZex <26...@qq.com.invalid> 于2023年7月22日周六 13:37写道:
>
> > Agree with you. We should also consider what metadata information is
> > necessary to count in the catalog.
> >
> >
> > Best Regards
> > TaoZex
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:
> >                                                   "dev"
> >                                                                 <
> > gaojun2048@apache.org&gt;;
> > 发送时间:&nbsp;2023年7月22日(星期六) 上午10:17
> > 收件人:&nbsp;"dev"<dev@seatunnel.apache.org&gt;;
> >
> > 主题:&nbsp;[Proposal] Spark&amp;Flink compatible catalog
> >
> >
> >
> > *background*
> >
> > Due to the existence of commercial versions and open source web, it is
> > necessary to clearly know the data source table and the table structure
> > processed by each node when defining tasks. So the entire architecture
> > layer needs to use a catalog approach, and the previous description of
> > SeaTunnelRowType cannot meet the synchronization requirements.
> > At present, the Zeta engine has supported the creation of connectors using
> > the catalog method, but spark/link still uses the SeaTunnelRowType
> > interface, which requires the connector to implement both the Catalog and
> > SeaTunnelRowType interfaces in order to execute on different engines. This
> > will require a lot of work and is meaningless.
> > To solve this problem, we need spark/link to first support the Catalog
> > method, so that the new connector can only implement the Catalog interface,
> > while the old connector can gradually remove the SeaTunnelRowType
> > interface.
> >
> > --
> >
> > Best Regards
> >
> > ------------
> >
> > EricJoy2048
> > gaojun2048@gmail.com

Re: [Proposal] Spark&Flink compatible catalog

Posted by hailin0 <wa...@apache.org>.
+1

Agree with you

TaoZex <26...@qq.com.invalid> 于2023年7月22日周六 13:37写道:

> Agree with you. We should also consider what metadata information is
> necessary to count in the catalog.
>
>
> Best Regards
> TaoZex
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
>                                                   "dev"
>                                                                 <
> gaojun2048@apache.org&gt;;
> 发送时间:&nbsp;2023年7月22日(星期六) 上午10:17
> 收件人:&nbsp;"dev"<dev@seatunnel.apache.org&gt;;
>
> 主题:&nbsp;[Proposal] Spark&amp;Flink compatible catalog
>
>
>
> *background*
>
> Due to the existence of commercial versions and open source web, it is
> necessary to clearly know the data source table and the table structure
> processed by each node when defining tasks. So the entire architecture
> layer needs to use a catalog approach, and the previous description of
> SeaTunnelRowType cannot meet the synchronization requirements.
> At present, the Zeta engine has supported the creation of connectors using
> the catalog method, but spark/link still uses the SeaTunnelRowType
> interface, which requires the connector to implement both the Catalog and
> SeaTunnelRowType interfaces in order to execute on different engines. This
> will require a lot of work and is meaningless.
> To solve this problem, we need spark/link to first support the Catalog
> method, so that the new connector can only implement the Catalog interface,
> while the old connector can gradually remove the SeaTunnelRowType
> interface.
>
> --
>
> Best Regards
>
> ------------
>
> EricJoy2048
> gaojun2048@gmail.com

RE:[Proposal] Spark&Flink compatible catalog

Posted by TaoZex <26...@qq.com.INVALID>.
Agree with you. We should also consider what metadata information is necessary to count in the catalog.


Best Regards
TaoZex
------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <gaojun2048@apache.org&gt;;
发送时间:&nbsp;2023年7月22日(星期六) 上午10:17
收件人:&nbsp;"dev"<dev@seatunnel.apache.org&gt;;

主题:&nbsp;[Proposal] Spark&amp;Flink compatible catalog



*background*

Due to the existence of commercial versions and open source web, it is
necessary to clearly know the data source table and the table structure
processed by each node when defining tasks. So the entire architecture
layer needs to use a catalog approach, and the previous description of
SeaTunnelRowType cannot meet the synchronization requirements.
At present, the Zeta engine has supported the creation of connectors using
the catalog method, but spark/link still uses the SeaTunnelRowType
interface, which requires the connector to implement both the Catalog and
SeaTunnelRowType interfaces in order to execute on different engines. This
will require a lot of work and is meaningless.
To solve this problem, we need spark/link to first support the Catalog
method, so that the new connector can only implement the Catalog interface,
while the old connector can gradually remove the SeaTunnelRowType interface.

-- 

Best Regards

------------

EricJoy2048
gaojun2048@gmail.com