You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/07/27 00:20:16 UTC
[GitHub] [incubator-seatunnel] ashulin opened a new issue, #2274: [ST-Engine][Checkpoint] The design of Checkpoint
ashulin opened a new issue, #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Backgroud & Motivation
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin closed issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
ashulin closed issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
URL: https://github.com/apache/incubator-seatunnel/issues/2274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1197940156
> @EricJoy2048 Now how do we deal with the union of multiple upstream operators? That's the same thing as the following <img alt="image" width="343" src="https://user-images.githubusercontent.com/36807946/181452486-e150e98c-7af7-425c-8afd-e53752677c99.png"> Users need `union (all)` post-processing, and we cannot separate them
We should create a special transform `union`. It can union all upstream data. Then `union` will have parallelism parameter, take this picture as an example, if parallelism = 8 (4+4), just send data to next action, if parallelism != 8, we should use queue( like Partition Transform) to change parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1197924787
> @EricJoy2048 Now how do we deal with the union of multiple upstream operators? That's the same thing as the following <img alt="image" width="343" src="https://user-images.githubusercontent.com/36807946/181452486-e150e98c-7af7-425c-8afd-e53752677c99.png"> Users need `union (all)` post-processing, and we cannot separate them
@Hisoka-X
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1197744571
> > <img alt="image" width="835" src="https://user-images.githubusercontent.com/32193458/181148775-705332e2-d997-4a50-9cd9-aa904be3af3d.png">
> > Can it be like this?
> > <img alt="image" width="881" src="https://user-images.githubusercontent.com/32193458/181149570-4180f833-d399-4ac0-88d9-7c6428aed989.png">
>
> No, it's different. The above writer has multiple input subtasks, which is to reduce the resource usage on the sink side;
From this design https://github.com/apache/incubator-seatunnel/issues/2261 we can know. Now, we only support `Source` and `PartitionTransform` operator to set parallelism, The parallelism of other connectors is equal to the sum of the parallelism of its upstream operators. So, in this case the parallelism of the sink operator is 3(One source have 2 parallelism and another one have 1 parallelism) not 2. Then, we have 3 source subtask and 3 sink subtask, I think it can replace to tow pipeline.
Or, do we have any other more suitable design for parallelism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1196382195
> <img alt="image" width="835" src="https://user-images.githubusercontent.com/32193458/181148775-705332e2-d997-4a50-9cd9-aa904be3af3d.png">
>
> Can it be like this?
>
> <img alt="image" width="881" src="https://user-images.githubusercontent.com/32193458/181149570-4180f833-d399-4ac0-88d9-7c6428aed989.png">
No, it's different. The above writer has multiple input subtasks, which is to reduce the resource usage on the sink side;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1196204323
<img width="835" alt="image" src="https://user-images.githubusercontent.com/32193458/181148775-705332e2-d997-4a50-9cd9-aa904be3af3d.png">
Can it be like this?
<img width="881" alt="image" src="https://user-images.githubusercontent.com/32193458/181149570-4180f833-d399-4ac0-88d9-7c6428aed989.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #2274: [ST-Engine][Checkpoint] The design of Checkpoint
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #2274:
URL: https://github.com/apache/incubator-seatunnel/issues/2274#issuecomment-1197797926
@EricJoy2048 Now how do we deal with the union of multiple upstream operators? That's the same thing as the following
<img width="343" alt="image" src="https://user-images.githubusercontent.com/36807946/181452486-e150e98c-7af7-425c-8afd-e53752677c99.png">
Users need `union (all)` post-processing, and we cannot separate them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org