You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by Lucifer Tyrant <ty...@gmail.com> on 2022/08/11 06:35:23 UTC
Discuss how to define data format in source conncector config
Hi seatunnel community,
According to pr #2392, seatunnel has a new feature to parse schema
information. Next I want to discuss with your guys how to define data
format in source connector configuration.
In my opinion, the config can be defined as the following:
schema {
fields {}
format = xxx
delimiter = xxx
}
We can abstract a new interface to parse format information in config. This
interface can use *seatunnel-formats* converting upstream from row data to
a seatunnel row. What do you think about it?
TyrantLucifer
Re: Discuss how to define data format in source conncector config
Posted by Lucifer Tyrant <ty...@gmail.com>.
I need to familiarize with the design first, before I do this work I will
discuss with your guys again.
BR,
TyrantLucifer
Zongwen Li <zo...@gmail.com> 于2022年8月12日周五 10:42写道：
> These factories should be enabled to allow users to configure more easily
> and support more features;
>
> Below I briefly explain the factory:
> 1. CatalogFactory: a feature with the same target as the user-defined
> schema, used to obtain the structured information of the table;
> 2. DeserializationFormatFactory & SerializationFormatFactory: used to
> support kafka, file and other unstructured connectors to process data in
> various formats;
> 3. TableSourceFactory: It is used to support multi-table reading and
> metadata column features. It is an upper-level extension of source:
> 4. TableSinkFactory: The same level as TableSourceFactory, but there are
> currently no supported features;
>
> You can review the flowchart in this proposal and refine the entire
> process: https://github.com/apache/incubator-seatunnel/issues/1608
>
> Best,
> Zongwen Li
>
> JUN GAO <ga...@apache.org> 于2022年8月12日周五 10:24写道：
>
> > Should we enable `org.apache.seatunnel.api.table.factory` and optimize
> the
> > design ? I think @Lucifer Tyrant can do this.
> >
> >
> > Zongwen Li <zo...@gmail.com> 于2022年8月12日周五 10:15写道：
> >
> > > The Format feature already exists, the problem is that we haven't
> enabled
> > > it yet;
> > > In fact, all the features in the
> `org.apache.seatunnel.api.table.factory`
> > > package under the `seatunnel-api` module are not enabled, they are
> > mutually
> > > dependent;
> > >
> > > Best,
> > > Zongwen Li
> > >
> > > 范佳 <fa...@qq.com.invalid> 于2022年8月11日周四 15:32写道：
> > >
> > > > +1
> > > >
> > > > Thanks for open this discussion.
> > > >
> > > > ________________________
> > > >
> > > > Jia Fan
> > > >
> > > > > 2022年8月11日 14:35，Lucifer Tyrant <ty...@gmail.com> 写道：
> > > > >
> > > > > Hi seatunnel community,
> > > > >
> > > > > According to pr #2392, seatunnel has a new feature to parse schema
> > > > > information. Next I want to discuss with your guys how to define
> data
> > > > > format in source connector configuration.
> > > > >
> > > > > In my opinion, the config can be defined as the following:
> > > > >
> > > > > schema {
> > > > > fields {}
> > > > > format = xxx
> > > > > delimiter = xxx
> > > > > }
> > > > >
> > > > > We can abstract a new interface to parse format information in
> > config.
> > > > This
> > > > > interface can use *seatunnel-formats* converting upstream from row
> > data
> > > > to
> > > > > a seatunnel row. What do you think about it?
> > > > >
> > > > > TyrantLucifer
> > > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > Best Regards
> >
> > ------------
> >
> > EricJoy2048
> > gaojun2048@gmail.com
> >
>
Re: Discuss how to define data format in source conncector config
Posted by Zongwen Li <zo...@gmail.com>.
These factories should be enabled to allow users to configure more easily
and support more features;
Below I briefly explain the factory:
1. CatalogFactory: a feature with the same target as the user-defined
schema, used to obtain the structured information of the table;
2. DeserializationFormatFactory & SerializationFormatFactory: used to
support kafka, file and other unstructured connectors to process data in
various formats;
3. TableSourceFactory: It is used to support multi-table reading and
metadata column features. It is an upper-level extension of source:
4. TableSinkFactory: The same level as TableSourceFactory, but there are
currently no supported features;
You can review the flowchart in this proposal and refine the entire
process: https://github.com/apache/incubator-seatunnel/issues/1608
Best,
Zongwen Li
JUN GAO <ga...@apache.org> 于2022年8月12日周五 10:24写道：
> Should we enable `org.apache.seatunnel.api.table.factory` and optimize the
> design ? I think @Lucifer Tyrant can do this.
>
>
> Zongwen Li <zo...@gmail.com> 于2022年8月12日周五 10:15写道：
>
> > The Format feature already exists, the problem is that we haven't enabled
> > it yet;
> > In fact, all the features in the `org.apache.seatunnel.api.table.factory`
> > package under the `seatunnel-api` module are not enabled, they are
> mutually
> > dependent;
> >
> > Best,
> > Zongwen Li
> >
> > 范佳 <fa...@qq.com.invalid> 于2022年8月11日周四 15:32写道：
> >
> > > +1
> > >
> > > Thanks for open this discussion.
> > >
> > > ________________________
> > >
> > > Jia Fan
> > >
> > > > 2022年8月11日 14:35，Lucifer Tyrant <ty...@gmail.com> 写道：
> > > >
> > > > Hi seatunnel community,
> > > >
> > > > According to pr #2392, seatunnel has a new feature to parse schema
> > > > information. Next I want to discuss with your guys how to define data
> > > > format in source connector configuration.
> > > >
> > > > In my opinion, the config can be defined as the following:
> > > >
> > > > schema {
> > > > fields {}
> > > > format = xxx
> > > > delimiter = xxx
> > > > }
> > > >
> > > > We can abstract a new interface to parse format information in
> config.
> > > This
> > > > interface can use *seatunnel-formats* converting upstream from row
> data
> > > to
> > > > a seatunnel row. What do you think about it?
> > > >
> > > > TyrantLucifer
> > > >
> > >
> > >
> >
>
>
> --
>
> Best Regards
>
> ------------
>
> EricJoy2048
> gaojun2048@gmail.com
>
Re: Discuss how to define data format in source conncector config
Posted by JUN GAO <ga...@apache.org>.
Should we enable `org.apache.seatunnel.api.table.factory` and optimize the
design ? I think @Lucifer Tyrant can do this.
Zongwen Li <zo...@gmail.com> 于2022年8月12日周五 10:15写道：
> The Format feature already exists, the problem is that we haven't enabled
> it yet;
> In fact, all the features in the `org.apache.seatunnel.api.table.factory`
> package under the `seatunnel-api` module are not enabled, they are mutually
> dependent;
>
> Best,
> Zongwen Li
>
> 范佳 <fa...@qq.com.invalid> 于2022年8月11日周四 15:32写道：
>
> > +1
> >
> > Thanks for open this discussion.
> >
> > ________________________
> >
> > Jia Fan
> >
> > > 2022年8月11日 14:35，Lucifer Tyrant <ty...@gmail.com> 写道：
> > >
> > > Hi seatunnel community,
> > >
> > > According to pr #2392, seatunnel has a new feature to parse schema
> > > information. Next I want to discuss with your guys how to define data
> > > format in source connector configuration.
> > >
> > > In my opinion, the config can be defined as the following:
> > >
> > > schema {
> > > fields {}
> > > format = xxx
> > > delimiter = xxx
> > > }
> > >
> > > We can abstract a new interface to parse format information in config.
> > This
> > > interface can use *seatunnel-formats* converting upstream from row data
> > to
> > > a seatunnel row. What do you think about it?
> > >
> > > TyrantLucifer
> > >
> >
> >
>
--
Best Regards
------------
EricJoy2048
gaojun2048@gmail.com
Re: Discuss how to define data format in source conncector config
Posted by Zongwen Li <zo...@gmail.com>.
The Format feature already exists, the problem is that we haven't enabled
it yet;
In fact, all the features in the `org.apache.seatunnel.api.table.factory`
package under the `seatunnel-api` module are not enabled, they are mutually
dependent;
Best,
Zongwen Li
范佳 <fa...@qq.com.invalid> 于2022年8月11日周四 15:32写道：
> +1
>
> Thanks for open this discussion.
>
> ________________________
>
> Jia Fan
>
> > 2022年8月11日 14:35，Lucifer Tyrant <ty...@gmail.com> 写道：
> >
> > Hi seatunnel community,
> >
> > According to pr #2392, seatunnel has a new feature to parse schema
> > information. Next I want to discuss with your guys how to define data
> > format in source connector configuration.
> >
> > In my opinion, the config can be defined as the following:
> >
> > schema {
> > fields {}
> > format = xxx
> > delimiter = xxx
> > }
> >
> > We can abstract a new interface to parse format information in config.
> This
> > interface can use *seatunnel-formats* converting upstream from row data
> to
> > a seatunnel row. What do you think about it?
> >
> > TyrantLucifer
> >
>
>
Re: Discuss how to define data format in source conncector config
Posted by 范佳 <fa...@qq.com.INVALID>.
+1
Thanks for open this discussion.
________________________
Jia Fan
> 2022年8月11日 14:35，Lucifer Tyrant <ty...@gmail.com> 写道：
>
> Hi seatunnel community,
>
> According to pr #2392, seatunnel has a new feature to parse schema
> information. Next I want to discuss with your guys how to define data
> format in source connector configuration.
>
> In my opinion, the config can be defined as the following:
>
> schema {
> fields {}
> format = xxx
> delimiter = xxx
> }
>
> We can abstract a new interface to parse format information in config. This
> interface can use *seatunnel-formats* converting upstream from row data to
> a seatunnel row. What do you think about it?
>
> TyrantLucifer
>