You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/19 07:48:25 UTC
[GitHub] [dolphinscheduler] EricGao888 opened a new issue, #11046: [Feature][Worker] Add cache mechanism in worker for resources
EricGao888 opened a new issue, #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
- Currently in DS, a worker executing tasks downloads related resources to local and cleans them up when tasks completed. We could improve the performance by caching frequently-used resources.
https://github.com/apache/dolphinscheduler/blob/ff34393333c076da74ef029cb4c60cf551f76b37/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L170-L182
https://github.com/apache/dolphinscheduler/blob/ff34393333c076da74ef029cb4c60cf551f76b37/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L241-L273
### Use case
* If some resources such as jars / tarballs are frequently used, they could be cached on workers and won't be downloaded every time when executing related tasks.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1189688220
> > > @ruanwenjun @zhongjiajie @caishunfeng @SbloodyS WDYT? May I ask what potential risks we may face if implementing such feature?
> >
> >
> > If the user modifies the content in different tasks, the cache will cause abnormal task execution.
>
> Yes, it's not a good idea to do this cache, we need to make the data consistent with the source.
@SbloodyS @ruanwenjun What about we add a button to each resource in resource center to let users choose to enable cache or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1188720648
@ruanwenjun @zhongjiajie @caishunfeng @SbloodyS WDYT? May I ask what potential risks we may face if implementing such feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1188722902
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1195378570
> > @SbloodyS @ruanwenjun What about we add a button / checkbox to each resource in resource center to let users choose to enable cache or not?
>
> In this case, we just throw the potential error to users, of cause we can show some hint when user try to change those resource with cache. But as I know @SbloodyS 's soc topic is to remove `t_da_resource` table and fetch resource list directly from storage system. After we remove the table, we can not add addition for resource file. I personally do not think it is a good idea to cache some resource in dolphinscheduler, it will add some raise and is it not the main function of task scheduling system should consider
Ok, this makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1188728523
> @ruanwenjun @zhongjiajie @caishunfeng @SbloodyS WDYT? May I ask what potential risks we may face if implementing such feature?
If the user modifies the content in different tasks, the cache will cause abnormal task execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1195364503
> @SbloodyS @ruanwenjun What about we add a button / checkbox to each resource in resource center to let users choose to enable cache or not?
In this case, we just throw the potential error to users, of cause we can show some hint when user try to change those resource with cache. But as I know @SbloodyS 's soc topic is to remove `t_da_resource` table and fetch resource list directly from storage system. After we remove the table, we can not add addition for resource file.
I personally do not think it is a good idea to cache some resource in dolphinscheduler, it will add some raise and is it not the main function of task scheduling system should consider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11046:
URL: https://github.com/apache/dolphinscheduler/issues/11046#issuecomment-1189057373
> > @ruanwenjun @zhongjiajie @caishunfeng @SbloodyS WDYT? May I ask what potential risks we may face if implementing such feature?
>
> If the user modifies the content in different tasks, the cache will cause abnormal task execution.
Yes, it's not a good idea to do this cache, we need to make the data consistent with the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 closed issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
Posted by GitBox <gi...@apache.org>.
EricGao888 closed issue #11046: [Feature][Worker] Add cache mechanism in worker for resources
URL: https://github.com/apache/dolphinscheduler/issues/11046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org