[GitHub] [druid] jasonk000 commented on issue #8801: KubernetesTaskRunner for running druid tasks as kubernetes jobs

[GitBox <gi...@apache.org>]

jasonk000 commented on issue #8801:
URL: https://github.com/apache/druid/issues/8801#issuecomment-793255773


   Ideally the `K8sTaskRunner` becomes an extension point where we can plug in any sort of `TaskRunner` extension. Is this the intended approach?
   
   I think there would be value in looking at this for a different job scheduler implementations (non-k8s, similar in some ideas). Having this as a standardised extension point will allow us to run this on any scheduler.
   
   With regards to comments on HPA / scaling requirements, currently we set task properties in Druid ingestion spec incl `context.druid.indexer.runner.javaOpts` to set thread counts and offheap memory limits, etc. I think we can use a similar mechanism in the ingestion spec to pass specific configuration to the task runner such as the expected resources required, with some defaults provided in configuration. This keeps the spec simple and can be modified for each use case.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org