[GitHub] QiuMM opened a new pull request #7066: Support assign tasks to run on different tiers of MiddleManagers

[GitBox <gi...@apache.org>]

QiuMM opened a new pull request #7066: Support assign tasks to run on different tiers of MiddleManagers
URL: https://github.com/apache/incubator-druid/pull/7066
 
 
   ## Motivation
   
   - Tasks with different task type might need different resource to run, i.e realtime task need much CPU and memory than Hadoop batch task and kill task. If all the MiddleManager workers share the same resource configuration will cause much waste.
   
   - Support cross data center, different datasource‘s task might want to run on different data center.
   
   - Current worker select strategy use `affinityConfig` to specify different datasource running on different workers. However, using this way we need to maintain the workers' ip and port information, this is bad since the ip and port might change, especially if we run MiddleManagers on cloud.
   
   ## Solution
   
   Separating MiddleManager nodes into tiers, then we can run tasks in different tiers based on tasks' task type and datasource name. First, add a tier property for the MiddleManager worker. Then, implement `WorkerSelectStrategy` to support assign tasks to run on different tiers. In this PR, providing two select strategy which naming `EqualDistributionWithTierSpecWorkerSelectStrategy`  and `FillCapacityWithTierSpecWorkerSelectStrategy` . You can review the docs in this PR to know the details of usage.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org