You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/09/02 11:13:31 UTC
[GitHub] [incubator-seatunnel] EricJoy2048 opened a new issue, #2612: [Improvement][Doc] Unify some contents in source and sink connector documents
EricJoy2048 opened a new issue, #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
The connector has some very important features, such as `batch` and `stream` and `exact-once` of the source connector. `exact-once` of sink connector. But there is no uniform fixed format description for these in the connector-v2 document.
I suggest that we provide a fixed document template to standardize the description of these features.
Such as:
template for Source connector:
- [x] batch
- [ ] stream
- [x] exact-once
template for Sink connector:
- [x] exact-once
Do you have any suggestions for this?
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236056564
> For some unstructured data connectors, `format` is also important. Beacuse connectors use this parameter as a basis for identifying the upstream data format
I will add `format` in the connector that needs it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235435687
template for Source connector:
- [ ] parallelism
- [ ] format(csv、json、parquet、avro、orc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 2013650523 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
2013650523 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1240871696
> template for Source & Sink connector:
>
> * [ ] schema projection
What does this mean in general?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236074568
> I suggest we add a doc file named `key-features.md` to explain the technical terms. And then every connector document link to this file
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1241685769
> > template for Source & Sink connector:
> >
> > * [ ] schema projection
>
> What does this mean in general?
+ 1, `user-defined schema` is perhaps better; `projection pushdown` is SQL optimization terminology.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236072083
I suggest we add a doc file named `key-features.md` to explain the technical terms. And then every connector document link to this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235438631
template for Source & Sink connector:
- [ ] schema projection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236037984
> > template for Source connector:
> >
> > * [ ] parallelism
> > * [ ] format(csv、json、parquet、avro、orc)
>
> I think `format(csv、json、parquet、avro、orc)` is not a universal feature for Source connector. It is only used for connectors that can directly read or write data files.
For some unstructured data connectors, `format` is also important. Beacuse connectors use this parameter as a basis for identifying the upstream data format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235376653
exact-once --> exactly-once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235446533
> template for Source connector:
>
> * [ ] parallelism
> * [ ] format(csv、json、parquet、avro、orc)
I think `format(csv、json、parquet、avro、orc)` is not a universal feature for Source connector. It is only used for connectors that can directly read or and write data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235378351
> exact-once --> exactly-once
Ok, Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235374360
@2013650523 @laglangyue @TyrantLucifer @iture123 @hailin0 @legendtkl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhuangchong commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235813090
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236069213
> I am not found `Hive Source` Connector in the document. @TyrantLucifer @CalvinKirs
>
> <img alt="image" width="356" src="https://user-images.githubusercontent.com/32193458/188258712-3142fef4-44d2-428f-a1e0-fd56d437b332.png">
Because the file sink connector refactoring hive sink connector also needs to be refactored, and so I will submit this part of the code with the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
TyrantLucifer closed issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
URL: https://github.com/apache/incubator-seatunnel/issues/2612
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235458169
> > template for Source connector:
> >
> > * [ ] parallelism
> > * [ ] format(csv、json、parquet、avro、orc)
>
> I think `format(csv、json、parquet、avro、orc)` is not a universal feature for Source connector. It is only used for connectors that can directly read or and write data files.
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1236059250
I am not found `Hive Source` Connector in the document. @TyrantLucifer @CalvinKirs
<img width="356" alt="image" src="https://user-images.githubusercontent.com/32193458/188258712-3142fef4-44d2-428f-a1e0-fd56d437b332.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2612: [Discuss][Doc] Unify some contents in source and sink connector documents
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2612:
URL: https://github.com/apache/incubator-seatunnel/issues/2612#issuecomment-1235447349
> template for Source & Sink connector:
>
> * [ ] schema projection
You are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org