You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/26 04:19:22 UTC
[GitHub] [incubator-seatunnel] Hisoka-X opened a new issue, #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
Hisoka-X opened a new issue, #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
SeaTunnel engine will receive the logical plan sent by the client, and the engine needs to convert it into a physical plan that can be directly executed. Therefore, it is necessary to process the logical execution plan and generate a physical plan through conversion. The specific process is as follows:
1. Logical Plan

Received the logical plan, we need to remove redundant Actions, and verifying the Schema(Transform2 and Transform 5 should be same)
2. Execution Plan

While converting to an execution plan:
- Transforms need to be merged, and the basis for merging is whether the data will be split after the Transform.
- Convert Shuffle Action to Queue
- Add Enumerator and Committer
3. Pipeline

Pipeline is currently the same as ExecutionPlan. Because we didn't add the Cache module, it will be different later
4. Physical Plan

We will split the Pipeline into separate executable tasks according to the degree of parallelism
After this, can send task to task execution service. Then task can run normally.
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X closed issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
Posted by GitBox <gi...@apache.org>.
Hisoka-X closed issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
URL: https://github.com/apache/incubator-seatunnel/issues/2269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269#issuecomment-1210281884
> hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?
1. The Queue will create when use shuffle transform (at now call partition transform). Used for data shuffle and change parallelism
2. The cache feature you mentioned will be added later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 2013650523 commented on issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
Posted by GitBox <gi...@apache.org>.
2013650523 commented on issue #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269#issuecomment-1210259376
hi,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org