[GitHub] [druid] Fryuni commented on pull request #13156: Support for middle manager less druid, tasks launch as k8s jobs

[GitBox <gi...@apache.org>]

Fryuni commented on PR #13156:
URL: https://github.com/apache/druid/pull/13156#issuecomment-1278322121

   Hey, this is an awesome PR, thanks for all the work
   I am concerned about this part, though:
   
   > The KubernetesTaskRunner runs in the overlord process. When it has a request to launch a task, it goes to the K8sApi, grabs its own PodSpec (the overlord itself). Takes that podSpec, modifies the necessary attributes (eg: command, labels, env variables etc). Takes the task.json, compresses and base64 encodes it. Then launches a K8s Job.
   
   The resource use for overlords and tasks is vastly different, using the same podSpec for those would require reserving much more resources for the overlords and the tasks than necessary.
   
   One idea is that instead of reading from the overlord pod itself, it would be better to provide the name of a config map containing the podSpec.
   
   Another idea: the middle-managers have tiers, which we can use to allocate tasks to middle-managers of different tiers. Those tiers could be used to define different podSpecs for different tasks without changing the tasks API.
   
   Those changes can be done on a later PR. All in all, this PR is a huge step forward in making Druid more scalable for cloud environments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org