[GitHub] [dolphinscheduler] EricGao888 commented on issue #9337: [Feature][master] When deployed on k8s and yarn, dolphinscheduler supports worker elastic scaling

[GitBox <gi...@apache.org>]

EricGao888 commented on issue #9337:
URL: https://github.com/apache/dolphinscheduler/issues/9337#issuecomment-1369398490

   > In this Feature, I would like to know how the following three questions are designed
   > 
   > 1. For timed workflow, when will their pod be created?   Because api server takes a certain amount of time to create the pod, will the task start time be delayed
   > 2. For resource allocation, if PODS are created dynamically based on workflow or task, how do dolphinscheduler specify the number of cpus and memory required by each pod?   Is it specified when workflow is created?
   > 3. When deploying dolphinscheduler, whether dynamic worker creation is optional and whether the old deployment mode is retained, such as dolphinscheduler on k8s and dolphinscheduler on native k8s
   
   @hdygxsj Good questions.
   1. I think we could make use of `K8S HPA` with the number of tasks in `TaskPriorityQueue` as the metric. DS will scale the worker once the number of queue tasks reaches a threshold. In this way, when task get dispatched, the workers has already been running and there will be no latency.
   2. In the design of elastic worker, we could follow `one worker one pod` instead of dynamically creating pod based on task, which is another thing stated in #13316 `K8S Executor`.
   3. IMHO, we could try to make things, e.g. executor, extensible and pluggable. Users will be able to choose which to use. In this way, the design will be compatible. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org