You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "gjzhappy (via GitHub)" <gi...@apache.org> on 2023/04/19 11:17:54 UTC
[GitHub] [dolphinscheduler] gjzhappy opened a new issue, #13957: [负载均衡][Yarn] 任务调度的时候为何不新增‘Yarn’策略?
gjzhappy opened a new issue, #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
查看doc,目前只提供三种负载均衡(随机/平滑轮询/线性加权),为何不把‘Yarn’当作一种策略,根据Yarn Service API,Yarn是支持如shell脚本在内的很多服务调用的,即xxx on Yarn,所以,这一块的可扩展性还是很大的,请问是否有开发计划?
我的应用场景是这样的,我希望在调用shell脚本的时候,也能提交给yarn。
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy closed issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy closed issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
URL: https://github.com/apache/dolphinscheduler/issues/13957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1514558283
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Looking at the doc, there are currently only three types of load balancing (random/smooth round robin/linear weighted), why not use 'Yarn' as a strategy, according to the Yarn Service API, Yarn supports many service calls such as shell scripts Yes, that is xxx on Yarn, so the scalability of this area is still very large. Is there any development plan?
My application scenario is like this, I hope that when calling the shell script, it can also be submitted to yarn.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515636401
> Hi, @gjzhappy , i agree with you that we have to extend and support more scheduling strategies, there's still a lot of room for improvement. **HOWEVER**, i don't think YARN can be regarded as a scheduling strategy for DS's Master. Thus, maybe you can extend shell task and support to submit it on Yarn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515642284
Hi, @Radeity, thanks a lot for your reply.
About the scheduling strategies, what i thought is to replace the strategy with the Yarn api, let Yarn decide where to dispatch the task, but i'm not sure if it is feasible for DS.(this is yarn api doc: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html)
And you mentioned about extend shell task to support it on Yarn, and I'm very interest in this, i want to know how to make it through DS, can you show me some tips? thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515603654
Hi, @gjzhappy , i agree with you that we have to extend and support more scheduling strategies, there's still a lot of room for improvement. **HOWEVER**, i don't think YARN can be regarded as a scheduling strategy for DS's Master. Thus, maybe you can extend shell task and support to submit it on Yarn, in an more elegant way, it can achieve the same outcome as `KubernetesExecutor` in our roadMap :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy closed issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy closed issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
URL: https://github.com/apache/dolphinscheduler/issues/13957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1514558509
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515604909
In addition, use english to describe your issue is better :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515697140
> About the scheduling strategies, what i thought is to replace the strategy with the Yarn api, let Yarn decide where to dispatch the task, but i'm not sure if it is feasible for DS.
In my view, it's not suitable to hand over task dispatching to Yarn.
> And you mentioned about extend shell task to support it on Yarn, and I'm very interest in this, i want to know how to make it through DS, can you show me some tips? thanks again.
Whether extend shell task has to be further discussed, ideally, it can achieve the same outcome as K8S Executor in our roadmap. You can try to give some proposal based on your scenario :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515806156
>
Maybe i need to find out where to extend on DS worker next, thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gjzhappy commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "gjzhappy (via GitHub)" <gi...@apache.org>.
gjzhappy commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515774269
> In my view, it's not suitable to hand over task dispatching to Yarn.
Scheduler and App Manager are fundamental functions of Yarn, do I misunderstand something?
> Whether extend shell task has to be further discussed, ideally, it can achieve the same outcome as K8S Executor in our roadmap. You can try to give some proposal based on your scenario :D
In my scenario, some of the tasks in the workflow will run as script, bat, sh etc. And the real procedure will triggered by the script, which will cost much more time, cpu, gpu and memory. I wonder if these scripts can be dispatched on Yarn, cause with Yarn, it supports like gpu management. Or, do you have any suggestion on this requirement? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #13957: [Load Balance][Yarn] Why not add the 'Yarn' policy when scheduling tasks?
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #13957:
URL: https://github.com/apache/dolphinscheduler/issues/13957#issuecomment-1515789946
> In my view, it's not suitable to hand over task dispatching to Yarn.
DS master is responsible for **DISPATCHING** tasks to DS worker. DS worker will submit task on Yarn if task (eg. Spark, Flink) support yarn mode. You can use yarn api to extend and support shell task submit to Yarn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org