You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/25 05:26:46 UTC
[GitHub] [incubator-seatunnel] hailin0 opened a new issue, #3178: [WIP][Umbrella] SeaTunnel Transform API
hailin0 opened a new issue, #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Describe the proposal
# Backgroud #2678
Currently, the transform code is bound to the single engine and cannot be shared to other engine using.
I propose that we create `transform-v2` module to unify transform implement, like source and sink, it is decoupled from the engine and can run on different engines.
Furthermore, we can use the `translation` module to integrate transform to seatunnel, flink, spark engine execute.
In order to ensure seatunnel's positioning as a data integration platform and not introduce work beyond the plan, the `transform-v2` will only support UDF level data conversion, and And unsupported sql transform(because st-engine unsupported sql parse & analysis).
# Objective
- Supports running on different engines
- Supports update fields datatype & value & orders
- Supports delete\add fields
# Overall Design
The Transform base process contains:
- Transform implement
<img width="564" alt="image" src="https://user-images.githubusercontent.com/14371345/197688154-2f38b996-3a42-4443-af13-bde9b74978bd.png">
<img width="599" alt="image" src="https://user-images.githubusercontent.com/14371345/197688226-c7e80f59-369b-4e4c-980d-8b77cae3bd56.png">
- Transform translation layer
- Adapt to flink engine
- Adapt to spark engine
- Adapt to seatunnel engine
<img width="721" alt="image" src="https://user-images.githubusercontent.com/14371345/197688271-8146086f-2b03-48ff-84b4-b4350998b0c6.png">
### Task list
Translation layer
- [ ] #3145
- [ ] Support transform-v2 for flink
- [ ] Support transform-v2 for spark
Transform
- [ ] Substring transform
- [ ] Convert date & time & timestamp transform
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1290307896
> transform method use function ? can support sql ?
@yuangjiang
Transform directly operates `stream<row>` on engines, currently unsupported using sql, but can achieve the same features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hk-lrzy commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
hk-lrzy commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1294702313
@hailin0 Can we describe the releationship betweens transform
like transform1 & transform2 and parallel, and transform3 use both transform1 & transform2 to do the filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1299516193
> @hailin0 Can we describe the releationship betweens transform like transform1 & transform2 and parallel, and transform3 use both transform1 & transform2 to do the filter.
reference
https://seatunnel.apache.org/docs/concept/config#other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] iture123 commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
iture123 commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1290333894
I suggest support send dirty data to the extra Sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3178: [Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1347569558
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1292167309
> I suggest support send dirty data to the extra Sink.
Good idea. This is another features -- data partition (selected data row will be send to specified sink)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3178: [Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1335982579
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yuangjiang commented on issue #3178: [WIP][Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
yuangjiang commented on issue #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178#issuecomment-1290263652
transform method use function ? can support sql ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3178: [Umbrella] SeaTunnel Transform V2 Design
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #3178: [Umbrella] SeaTunnel Transform V2 Design
URL: https://github.com/apache/incubator-seatunnel/issues/3178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org