You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/11/10 03:01:39 UTC
[GitHub] [dolphinscheduler] Radeity opened a new issue, #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Radeity opened a new issue, #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
DS supports file transfer between tasks in this PR:
- https://github.com/apache/dolphinscheduler/pull/12552
<br>
In current way, intermediate files will be uploaded from `srcPath` in upstream task to `sourcePath` in resource center, and then downstream task will load the file from `sourcePath` in resource center.
#### Current logic
<img width="1495" alt="image" src="https://user-images.githubusercontent.com/45198818/200986807-ac152753-658f-4925-99f5-8c3c5fd15d57.png">
<br>
However, it's unnecessary to upload files to resource center, rather, we can transfer files end-to-end from upstream worker to downstream worker by `scp`.
In detail, upstream worker sends scp command template to resource center instead of raw file, also save in resourcePath. Then, downstream worker reads the command template from resource center and complete the command with targetPath. Finally, downstream worker can read file from upstream worker by executing `scp` command.
<img width="1559" alt="image" src="https://user-images.githubusercontent.com/45198818/200986035-3208b7d6-99b7-4dc1-bcd9-73427ea99dc2.png">
<br>
BTW, i won't replace current way, but add two user options:
- `transfer.file.dir`: tmp directory to save intermediate files in worker node.
- `transfer.file.size`: maximum limits of tmp directory. Intermediate file will be uploaded if exceed the limit, otherwise, just upload command template. We can add some flag and the downstream worker will know whether read the whole file or have to execute `scp` command.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1416884432
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
Radeity commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1309732291
PTAL. @jieguangzhou @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
Radeity commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1371838444
Reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] closed issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
URL: https://github.com/apache/dolphinscheduler/issues/12849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1459075816
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1345416785
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] closed issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
URL: https://github.com/apache/dolphinscheduler/issues/12849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1588285026
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1309702237
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1356917943
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #12849: [Improvement][Task] Support end-to-end transfer file between tasks
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12849:
URL: https://github.com/apache/dolphinscheduler/issues/12849#issuecomment-1622728711
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org