You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/08/19 02:25:44 UTC
[GitHub] [incubator-seatunnel] CalvinKirs opened a new issue, #2469: [Connector]Extract Connector public fields
CalvinKirs opened a new issue, #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469
Connector has a public field fields, usually the schema that defines the Connector,
But at present our definitions are very confusing, each Connector has its own set, a good example is this,
https://github.com/apache/incubator-seatunnel/pull/2439/files
https://github.com/apache/incubator-seatunnel/pull/2451/files
At the same time, there is also an abstraction of public method extraction, which allows developers to not need to parse the schema again, etc.
But I have questions about the fields and schemas in it, can we unify a set of standards? The latter use this.
In addition, for some fields (used by some connectors), such as delimiter, format, etc., whether the standard and format can be unified (I am not sure), we may need a package to do these things,
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2469: [Connector]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220198862
In my opinion, `format` in file connectors is a little ambiguous. For some text-based connector `format` is a best option. But your question is very right, we should unify a set of standards for all connectors to support user-defined schema. I think we can defined some common parameters to represent the current schema information.
`fields` represent the fields information of upstream data
`format` represent the format of upstream data, for example: `json` `csv` `orc` `parquet` etc...
`delimiter` represent the delimiter of upstream text data to splite fields.
The function of parsing user-defined schema has implemented in `connector-common`, all that remains is that unified the parameters of connectors
What do you think about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1221522727
`format` is useless in some connectors, such as mysql, oracle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2469: [Connector]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220221860
This discuss is base on Connector-V2 or Connector?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220265231
> > In my opinion, `format` in file connectors is a little ambiguous. For some text-based connector `format` is a best option. But your question is very right, we should unify a set of standards for all connectors to support user-defined schema. I think we can defined some common parameters to represent the current schema information.
> >
> > `fields` represent the fields information of upstream data
> >
> > `format` represent the format of upstream data, for example: `json` `csv` `orc` `parquet` etc...
> >
> > `delimiter` represent the delimiter of upstream text data to splite fields.
> >
> > The function of parsing user-defined schema has implemented in `connector-common`, all that remains is that unified the parameters of connectors
> >
> > What do you think about?
>
> great, but about `fields` and `schema`, do we still need nesting? For now, it doesn't seem necessary.
The same as you, i think nesting is useless too. So the conclusion is that we will unified the schema options for all connectors as behind, right? I can do this work and I will create a issue to track this work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
TyrantLucifer closed issue #2469: [Connector-v2]Extract Connector public fields
URL: https://github.com/apache/incubator-seatunnel/issues/2469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1221521683
I don't think `format` should be a child of `schema`, `format` should be a connector-level option with `schema`.
`schema` is used to represent the meta information of the table such as field information(`fields`), primary key, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220308858
agree with you. you don't need to create an issue separately, you can do it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220225825
> In my opinion, `format` in file connectors is a little ambiguous. For some text-based connector `format` is a best option. But your question is very right, we should unify a set of standards for all connectors to support user-defined schema. I think we can defined some common parameters to represent the current schema information.
>
> `fields` represent the fields information of upstream data
>
> `format` represent the format of upstream data, for example: `json` `csv` `orc` `parquet` etc...
>
> `delimiter` represent the delimiter of upstream text data to splite fields.
>
> The function of parsing user-defined schema has implemented in `connector-common`, all that remains is that unified the parameters of connectors
>
> What do you think about?
great, but about `fields` and `schema`, do we still need nesting? For now, it doesn't seem necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220225121
> This discuss is base on Connector-V2 or Connector?
yup, I updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2469: [Connector-v2]Extract Connector public fields
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2469:
URL: https://github.com/apache/incubator-seatunnel/issues/2469#issuecomment-1220314153
The sub tasks about this issue as the following shown:
- [ ] local file source connector
- [ ] hdfs file source connector
- [ ] oss file source connector
- [ ] http source connector
- [ ] kafka source connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org