You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/26 04:19:22 UTC

[GitHub] [incubator-seatunnel] Hisoka-X opened a new issue, #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

Hisoka-X opened a new issue, #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269

   ### Search before asking
   
   - [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   SeaTunnel engine will receive the logical plan sent by the client, and the engine needs to convert it into a physical plan that can be directly executed. Therefore, it is necessary to process the logical execution plan and generate a physical plan through conversion. The specific process is as follows:
   1. Logical Plan
   ![image](https://user-images.githubusercontent.com/32387433/180914738-b21e96e6-e1ea-4947-bd70-549415da44be.png)
   Received the logical plan, we need to remove redundant Actions, and verifying the Schema(Transform2 and Transform 5 should be same)
   2. Execution Plan
   ![image](https://user-images.githubusercontent.com/32387433/180918178-68f0dd22-b8d6-4347-ab62-f31bd28e8678.png)
   While converting to an execution plan: 
   - Transforms need to be merged, and the basis for merging is whether the data will be split after the Transform. 
   - Convert Shuffle Action to Queue
   - Add Enumerator and Committer
   3. Pipeline
   ![image](https://user-images.githubusercontent.com/32387433/180918235-738d52b0-3209-4cb9-a7cb-f833368f4b8f.png)
   Pipeline is currently the same as ExecutionPlan. Because we didn't add the Cache module, it will be different later
   4. Physical Plan
   ![image](https://user-images.githubusercontent.com/32387433/180918285-6faa7ba5-1bd0-4515-9b4a-5c0a8e65b913.png)
   We will split the Pipeline into separate executable tasks according to the degree of parallelism
   After this, can send task to task execution service. Then task can run normally.
   
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-seatunnel] Hisoka-X closed issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

Posted by GitBox <gi...@apache.org>.
Hisoka-X closed issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan
URL: https://github.com/apache/incubator-seatunnel/issues/2269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269#issuecomment-1210281884

   > hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?
   
   1. The Queue will create when use shuffle transform (at now call partition transform). Used for data shuffle and change parallelism
   2. The cache feature you mentioned will be added later


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-seatunnel] 2013650523 commented on issue #2269: [ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

Posted by GitBox <gi...@apache.org>.
2013650523 commented on issue #2269:
URL: https://github.com/apache/incubator-seatunnel/issues/2269#issuecomment-1210259376

   hi,Let me ask you a question, Why is the data queue designed to decouple source and sink?  Is the data cached in the queue when the source fails? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org